Skip to content
Find file
Fetching contributors…
Cannot retrieve contributors at this time
139 lines (120 sloc) 8.13 KB
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<title>Getting Started</title>
<body><style>
.codehilite .hll { background-color: #ffffcc }
.codehilite { background: #f0f0f0; }
.codehilite .c { color: #60a0b0; font-style: italic } /* Comment */
.codehilite .err { border: 1px solid #FF0000 } /* Error */
.codehilite .k { color: #007020; font-weight: bold } /* Keyword */
.codehilite .o { color: #666666 } /* Operator */
.codehilite .cm { color: #60a0b0; font-style: italic } /* Comment.Multiline */
.codehilite .cp { color: #007020 } /* Comment.Preproc */
.codehilite .c1 { color: #60a0b0; font-style: italic } /* Comment.Single */
.codehilite .cs { color: #60a0b0; background-color: #fff0f0 } /* Comment.Special */
.codehilite .gd { color: #A00000 } /* Generic.Deleted */
.codehilite .ge { font-style: italic } /* Generic.Emph */
.codehilite .gr { color: #FF0000 } /* Generic.Error */
.codehilite .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.codehilite .gi { color: #00A000 } /* Generic.Inserted */
.codehilite .go { color: #808080 } /* Generic.Output */
.codehilite .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
.codehilite .gs { font-weight: bold } /* Generic.Strong */
.codehilite .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.codehilite .gt { color: #0040D0 } /* Generic.Traceback */
.codehilite .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
.codehilite .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
.codehilite .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
.codehilite .kp { color: #007020 } /* Keyword.Pseudo */
.codehilite .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
.codehilite .kt { color: #902000 } /* Keyword.Type */
.codehilite .m { color: #40a070 } /* Literal.Number */
.codehilite .s { color: #4070a0 } /* Literal.String */
.codehilite .na { color: #4070a0 } /* Name.Attribute */
.codehilite .nb { color: #007020 } /* Name.Builtin */
.codehilite .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
.codehilite .no { color: #60add5 } /* Name.Constant */
.codehilite .nd { color: #555555; font-weight: bold } /* Name.Decorator */
.codehilite .ni { color: #d55537; font-weight: bold } /* Name.Entity */
.codehilite .ne { color: #007020 } /* Name.Exception */
.codehilite .nf { color: #06287e } /* Name.Function */
.codehilite .nl { color: #002070; font-weight: bold } /* Name.Label */
.codehilite .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
.codehilite .nt { color: #062873; font-weight: bold } /* Name.Tag */
.codehilite .nv { color: #bb60d5 } /* Name.Variable */
.codehilite .ow { color: #007020; font-weight: bold } /* Operator.Word */
.codehilite .w { color: #bbbbbb } /* Text.Whitespace */
.codehilite .mf { color: #40a070 } /* Literal.Number.Float */
.codehilite .mh { color: #40a070 } /* Literal.Number.Hex */
.codehilite .mi { color: #40a070 } /* Literal.Number.Integer */
.codehilite .mo { color: #40a070 } /* Literal.Number.Oct */
.codehilite .sb { color: #4070a0 } /* Literal.String.Backtick */
.codehilite .sc { color: #4070a0 } /* Literal.String.Char */
.codehilite .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
.codehilite .s2 { color: #4070a0 } /* Literal.String.Double */
.codehilite .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
.codehilite .sh { color: #4070a0 } /* Literal.String.Heredoc */
.codehilite .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
.codehilite .sx { color: #c65d09 } /* Literal.String.Other */
.codehilite .sr { color: #235388 } /* Literal.String.Regex */
.codehilite .s1 { color: #4070a0 } /* Literal.String.Single */
.codehilite .ss { color: #517918 } /* Literal.String.Symbol */
.codehilite .bp { color: #007020 } /* Name.Builtin.Pseudo */
.codehilite .vc { color: #bb60d5 } /* Name.Variable.Class */
.codehilite .vg { color: #bb60d5 } /* Name.Variable.Global */
.codehilite .vi { color: #bb60d5 } /* Name.Variable.Instance */
.codehilite .il { color: #40a070 } /* Literal.Number.Integer.Long */
</style>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-377063-3']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
<h1>Getting Started</h1>
<h2>Downloading</h2>
<p>Development versions of Skywriting are stored on <a href="http://github.com">Github</a>, with the main repository available <a href="http://github.com/mrry/skywriting">here</a>:</p>
<div class="codehilite"><pre>$ git checkout http://github.com/mrry/skywriting.git</pre></div>
<p>Check this repository out, and then compile and install it using PyPi:</p>
<div class="codehilite"><pre>$ python setup.py install</pre></div>
<p>This will install the Python libraries, as well as the helper scripts <code>sw-master</code>, <code>sw-worker</code> and <code>sw-job</code>, which are explained below.</p>
<p>TODO: java setup</p>
<h2>Concepts</h2>
<p>Skywriting uses a master/worker architecture, and so you will need to set up a single master instance, and as many workers as you have machines/cores in your cluster.
Communication between these instances happens using HTTP POST and a simple JSON RPC format.</p>
<h3>Master Setup</h3>
<p>The <code>sw-master</code> command sets up a master server listening on a port, using the fully-qualified domain name of the host.</p>
<div class="codehilite"><pre>$ sw-master [-p port]</pre></div>
<p>Note that it does not listen on <code>localhost</code> by default, and so you should use the FQDN to address the node.
It defaults to port <code>8080</code>, and do not forget to open up a hole in your firewall so that other worker nodes in the cluster can also see the master.</p>
<h3>Worker Setup</h3>
<p>Each worker runs on its own process and port, where it listens for jobs to be dispatched to it by the master process.
Workers need to be able to see the master port, and also to receive connections from the master server, so ensure your firewall is suitably configured.
Workers never directly communication with each other.</p>
<p>To get started with a single worker, type in:</p>
<div class="codehilite"><pre>$ sw-worker -m &lt;master uri&gt; -p &lt;port&gt;</pre></div>
<h3>Job Submission</h3>
<p>Now that you have a master and some workers, it is time to submit a job for parallel processing. There is a script to get you started in <code>src/sw/helloworld.sw</code>:</p>
<div class="codehilite"><pre>function hello(who) {
return &quot;hello &quot; + who;
}
greeting = spawn(hello,[&quot;world&quot;]);
return *greeting;</pre></div>
<p>This illustrates the Javascript-like syntax of Skywriting, and spawns a task which return a static greeting string.
Of course, you could return the string directly without the intermediate function, but using one lets us test that your master/worker setup is functioning correctly.</p>
<p>Submit the job by running:</p>
<div class="codehilite"><pre>$ sw-job -m &lt;master uri&gt; &lt;script file&gt;</pre></div>
<p>Currently a lot of debugging information gets emitted, this will be quietened as development settles down.</p>
<h3>Interactive Console</h3>
<p>There is also an interactive console you can use to evaluate Skywriting scripts directly. The <code>sw-console</code> script connects to a master, just like a worker does, and drops you into an interactive shell.</p>
<div class="codehilite"><pre>$ sw-console --master http://click.local:8080
(Cmd) function hello() { return &quot;hello world&quot;; }
&lt;mrry.mercator.runtime.references.SWNullReference instance at 0x101369b48&gt;
(Cmd) x=spawn(hello,[]);
&lt;mrry.mercator.runtime.references.SWNullReference instance at 0x1013698c0&gt;
(Cmd) return *x;
hello world</pre></div>
<p>The console itself is a wrapper to the job submission interface, so you still need to have workers connected to run the interactive commands as you type them in. If a worker is not available, the console will block until the job is completed.</p></body>
Something went wrong with that request. Please try again.