Skip to content

Commit

Permalink
first cut at installation guide
Browse files Browse the repository at this point in the history
  • Loading branch information
@avsm
avsm committed May 18, 2010
1 parent b3a0c03 commit 1c62887
Show file tree
Hide file tree
Showing 5 changed files with 191 additions and 1 deletion.
2 changes: 1 addition & 1 deletion Makefile
View file
@@ -1,4 +1,4 @@
FILES=index tutorial
FILES=installation tutorial index
.PHONY: all
all:
/anfs/www/tools/bin/ucampas $(FILES)
Expand Down
1 change: 1 addition & 0 deletions index-b.html
View file
Expand Up @@ -67,6 +67,7 @@
<h1>Skywriting</h1>
<p>Skywriting is a programming language and execution engine for distributed, parallel computation. Like previous frameworks (such as <a href="http://labs.google.com/papers/mapreduce.html">MapReduce</a> and <a href="http://labs.google.com/papers/mapreduce.html">Dryad</a>, it masks the complexity of distributed programming. Unlike those frameworks, it offers a full general-purpose programming language for expressing distributed coordination, which enables developers to implement algorithms using arbitrary task structures.</p>
<ul>
<li><a href="installation.html">Getting Started</a></li>
<li><a href="tutorial.html">Tutorial</a></li>
<li><a href="http://github.com/mrry/skywriting">Source code</a></li>
</ul>
Expand Down
1 change: 1 addition & 0 deletions index.md
View file
Expand Up @@ -3,6 +3,7 @@ Skywriting

Skywriting is a programming language and execution engine for distributed, parallel computation. Like previous frameworks (such as [MapReduce](http://labs.google.com/papers/mapreduce.html) and [Dryad](http://labs.google.com/papers/mapreduce.html), it masks the complexity of distributed programming. Unlike those frameworks, it offers a full general-purpose programming language for expressing distributed coordination, which enables developers to implement algorithms using arbitrary task structures.

* [Getting Started](installation.html)
* [Tutorial](tutorial.html)
* [Source code](http://github.com/mrry/skywriting)

Expand Down
118 changes: 118 additions & 0 deletions installation-b.html
View file
@@ -0,0 +1,118 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<title>Getting Started</title>
<body><style>
.codehilite .hll { background-color: #ffffcc }
.codehilite { background: #f0f0f0; }
.codehilite .c { color: #60a0b0; font-style: italic } /* Comment */
.codehilite .err { border: 1px solid #FF0000 } /* Error */
.codehilite .k { color: #007020; font-weight: bold } /* Keyword */
.codehilite .o { color: #666666 } /* Operator */
.codehilite .cm { color: #60a0b0; font-style: italic } /* Comment.Multiline */
.codehilite .cp { color: #007020 } /* Comment.Preproc */
.codehilite .c1 { color: #60a0b0; font-style: italic } /* Comment.Single */
.codehilite .cs { color: #60a0b0; background-color: #fff0f0 } /* Comment.Special */
.codehilite .gd { color: #A00000 } /* Generic.Deleted */
.codehilite .ge { font-style: italic } /* Generic.Emph */
.codehilite .gr { color: #FF0000 } /* Generic.Error */
.codehilite .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.codehilite .gi { color: #00A000 } /* Generic.Inserted */
.codehilite .go { color: #808080 } /* Generic.Output */
.codehilite .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
.codehilite .gs { font-weight: bold } /* Generic.Strong */
.codehilite .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.codehilite .gt { color: #0040D0 } /* Generic.Traceback */
.codehilite .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
.codehilite .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
.codehilite .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
.codehilite .kp { color: #007020 } /* Keyword.Pseudo */
.codehilite .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
.codehilite .kt { color: #902000 } /* Keyword.Type */
.codehilite .m { color: #40a070 } /* Literal.Number */
.codehilite .s { color: #4070a0 } /* Literal.String */
.codehilite .na { color: #4070a0 } /* Name.Attribute */
.codehilite .nb { color: #007020 } /* Name.Builtin */
.codehilite .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
.codehilite .no { color: #60add5 } /* Name.Constant */
.codehilite .nd { color: #555555; font-weight: bold } /* Name.Decorator */
.codehilite .ni { color: #d55537; font-weight: bold } /* Name.Entity */
.codehilite .ne { color: #007020 } /* Name.Exception */
.codehilite .nf { color: #06287e } /* Name.Function */
.codehilite .nl { color: #002070; font-weight: bold } /* Name.Label */
.codehilite .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
.codehilite .nt { color: #062873; font-weight: bold } /* Name.Tag */
.codehilite .nv { color: #bb60d5 } /* Name.Variable */
.codehilite .ow { color: #007020; font-weight: bold } /* Operator.Word */
.codehilite .w { color: #bbbbbb } /* Text.Whitespace */
.codehilite .mf { color: #40a070 } /* Literal.Number.Float */
.codehilite .mh { color: #40a070 } /* Literal.Number.Hex */
.codehilite .mi { color: #40a070 } /* Literal.Number.Integer */
.codehilite .mo { color: #40a070 } /* Literal.Number.Oct */
.codehilite .sb { color: #4070a0 } /* Literal.String.Backtick */
.codehilite .sc { color: #4070a0 } /* Literal.String.Char */
.codehilite .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
.codehilite .s2 { color: #4070a0 } /* Literal.String.Double */
.codehilite .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
.codehilite .sh { color: #4070a0 } /* Literal.String.Heredoc */
.codehilite .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
.codehilite .sx { color: #c65d09 } /* Literal.String.Other */
.codehilite .sr { color: #235388 } /* Literal.String.Regex */
.codehilite .s1 { color: #4070a0 } /* Literal.String.Single */
.codehilite .ss { color: #517918 } /* Literal.String.Symbol */
.codehilite .bp { color: #007020 } /* Name.Builtin.Pseudo */
.codehilite .vc { color: #bb60d5 } /* Name.Variable.Class */
.codehilite .vg { color: #bb60d5 } /* Name.Variable.Global */
.codehilite .vi { color: #bb60d5 } /* Name.Variable.Instance */
.codehilite .il { color: #40a070 } /* Literal.Number.Integer.Long */
</style>
<h1>Getting Started</h1>
<h2>Downloading</h2>
<p>Development versions of Skywriting are stored on <a href="http://github.com">Github</a>, with the main repository available <a href="http://github.com/mrry/skywriting">here</a>:</p>
<div class="codehilite"><pre><span class="nv">$ </span>git checkout http://github.com/mrry/skywriting.git
</pre></div>


<p>Check this repository out, and then compile and install it using PyPi:</p>
<div class="codehilite"><pre><span class="nv">$ </span>python setup.py install
</pre></div>


<p>This will install the Python libraries, as well as the helper scripts <code>sw-master</code>, <code>sw-worker</code> and <code>sw-job</code>, which are explained below.</p>
<p>TODO: java setup</p>
<h2>Concepts</h2>
<p>Skywriting uses a master/worker architecture, and so you will need to set up a single master instance, and as many workers as you have machines/cores in your cluster.
Communication between these instances happens using HTTP POST and a simple JSON RPC format.</p>
<h3>Master Setup</h3>
<p>The <code>sw-master</code> command sets up a master server listening on a port, using the fully-qualified domain name of the host.</p>
<div class="codehilite"><pre><span class="n">sw</span><span class="o">-</span><span class="n">master</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">port</span><span class="p">]</span>
</pre></div>


<p>Note that it does not listen on <code>localhost</code> by default, and so you should use the FQDN to address the node.
It defaults to port <code>8080</code>, and do not forget to open up a hole in your firewall so that other worker nodes in the cluster can also see the master.</p>
<h3>Worker Setup</h3>
<p>Each worker runs on its own process and port, where it listens for jobs to be dispatched to it by the master process.
Workers need to be able to see the master port, and also to receive connections from the master server, so ensure your firewall is suitably configured.
Workers never directly communication with each other.</p>
<p>To get started with a single worker, type in:</p>
<div class="codehilite"><pre><span class="n">sw</span><span class="o">-</span><span class="n">worker</span> <span class="o">-</span><span class="n">m</span> <span class="o">&lt;</span><span class="n">master</span> <span class="n">uri</span><span class="o">&gt;</span> <span class="o">-</span><span class="n">p</span> <span class="s-Regexp">&lt;port&gt;</span>
</pre></div>


<h3>Job Submission</h3>
<p>Now that you have a master and some workers, it is time to submit a job for parallel processing. There is a script to get you started in <code>src/sw/helloworld.sw</code>:</p>
<div class="codehilite"><pre><span class="kd">function</span> <span class="nx">hello</span><span class="p">(</span><span class="nx">who</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="s2">&quot;hello &quot;</span> <span class="o">+</span> <span class="nx">who</span><span class="p">;</span>
<span class="p">}</span>

<span class="nx">greeting</span> <span class="o">=</span> <span class="nx">spawn</span><span class="p">(</span><span class="nx">hello</span><span class="p">,[</span><span class="s2">&quot;world&quot;</span><span class="p">]);</span>
<span class="k">return</span> <span class="o">*</span><span class="nx">greeting</span><span class="p">;</span>
</pre></div>


<p>This illustrates the Javascript-like syntax of Skywriting, and spawns a task which return a static greeting string.
Of course, you could return the string directly without the intermediate function, but using one lets us test that your master/worker setup is functioning correctly.</p>
<p>Submit the job by running:</p>
<p>sw-job -m <master uri> <script file></p>
<p>Currently a lot of debugging information gets emitted, this will be quietened as development settles down.</p>
<h2>Interactive Console</h2>
<p>TODO: REPL information</p></body>
70 changes: 70 additions & 0 deletions installation.md
View file
@@ -0,0 +1,70 @@
Getting Started
===============

Downloading
-----------

Development versions of Skywriting are stored on [Github](http://github.com), with the main repository available [here](http://github.com/mrry/skywriting):

:::bash
$ git checkout http://github.com/mrry/skywriting.git

Check this repository out, and then compile and install it using PyPi:

:::bash
$ python setup.py install

This will install the Python libraries, as well as the helper scripts `sw-master`, `sw-worker` and `sw-job`, which are explained below.

TODO: java setup

Concepts
--------

Skywriting uses a master/worker architecture, and so you will need to set up a single master instance, and as many workers as you have machines/cores in your cluster.
Communication between these instances happens using HTTP POST and a simple JSON RPC format.

### Master Setup

The `sw-master` command sets up a master server listening on a port, using the fully-qualified domain name of the host.

sw-master [-p port]

Note that it does not listen on `localhost` by default, and so you should use the FQDN to address the node.
It defaults to port `8080`, and do not forget to open up a hole in your firewall so that other worker nodes in the cluster can also see the master.

### Worker Setup

Each worker runs on its own process and port, where it listens for jobs to be dispatched to it by the master process.
Workers need to be able to see the master port, and also to receive connections from the master server, so ensure your firewall is suitably configured.
Workers never directly communication with each other.

To get started with a single worker, type in:

sw-worker -m <master uri> -p <port>

### Job Submission

Now that you have a master and some workers, it is time to submit a job for parallel processing. There is a script to get you started in `src/sw/helloworld.sw`:

:::javascript
function hello(who) {
return "hello " + who;
}

greeting = spawn(hello,["world"]);
return *greeting;

This illustrates the Javascript-like syntax of Skywriting, and spawns a task which return a static greeting string.
Of course, you could return the string directly without the intermediate function, but using one lets us test that your master/worker setup is functioning correctly.

Submit the job by running:

sw-job -m <master uri> <script file>

Currently a lot of debugging information gets emitted, this will be quietened as development settles down.

Interactive Console
-------------------

TODO: REPL information

0 comments on commit 1c62887

Please sign in to comment.