Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 073c2964c8
Fetching contributors…

Cannot retrieve contributors at this time

182 lines (101 sloc) 7.34 kb
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
<html xmlns="" xml:lang="en" lang="en">
<link rel="stylesheet" href="stylesheets/screen.css" type="text/css" media="screen" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script src="javascripts/" type="text/javascript"></script>
<script type="text/javascript">
window.onload = function() {
settings = {
tl: { radius: 10 },
tr: { radius: 10 },
bl: { radius: 10 },
br: { radius: 10 },
antiAlias: true,
autoPad: true,
validTags: ["div"]
var versionBox = new curvyCorners(settings, document.getElementById("version"));
<div id="main">
<div id="version" class="clickable" onclick='document.location = ""; return false'>
<p>Get Version</p>
<a href="" class="numbers">0.9.3</a>
<p><img src="skynet_logo.jpg" alt="" /></p>
<h2>A Ruby MapReduce Framework</h2>
<p><a href="">Skynet RubyForge Home</a><br />
<a href="doc/index.html">Documentation</a></p>
<h2>See Skynet at RailsConf 08</h2>
<h4>Sat Mar 31, 2008 4:25pm &#8211; 5:15pm <span class="caps">PDT</span> @ Portland Ballroom 255</h4>
<p><a href="" target="_blank">More Info</a></p>
<p><a href="" target="_blank">
<img src="" width="728" height="90" border="0" alt="RailsConf 2008" title="RailsConf 2008" />
<h2>What is Skynet</h2>
<p>Skynet is an open source Ruby implementation of Google&#8217;s MapReduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers. If you&#8217;d like to learn more about MapReduce, see my intro at the bottom of this document.</p>
<p>Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a &#8220;peer recovery&#8221; system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special &#8216;master&#8217; servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.</p>
<p>For more detailed documentation see the following:</p>
<p>Skynet::Job &#8211; The main interface to Skynet; includes an example of how to use Skynet</p>
<p>Skynet::Config &#8211; Configuration options</p>
<p>bin/skynet[link:files/bin/skynet.html] &#8211; Starting Skynet</p>
<p>bin/skynet_install[link:files/bin/skynet_install.html] &#8211; Installing Skynet into a local project</p>
<p>There are also some examples in the examples/ directory included with Skynet.</p>
<p>Skynet can be installed via RubyGems:</p>
<pre><code>$ sudo gem install skynet</code></pre>
<p>or grab the bleeding edge skynet in svn at
$ svn checkout svn+ssh://
$ cd skynet; rake install_gem</p>
<h2>Initial Setup</h2>
<p>Skynet works by putting &#8220;tasks&#8221; on a message queue which are picked up by skynet workers. The workers execute tasks and put their results back on the message queue. Skynet workers need to load your code at startup in order to be able to execute your tasks. This loading is handled by installing a skynet config file into your app running skynet_install[link:files/bin/skynet_install.html]. </p>
<pre><code>$ skynet_install <a href="</del>-mysql"><del>-rails</a> APP_ROOT_DIR</code></pre>
<p>This creates a file called skynet_config.rb in <span class="caps">APP</span>_ROOT_DIR/config to which you can add the relevant requires. For example, you might have a rails app and want some of that code to run asynchronously or in a distributed way. Just run &#8216;skynet_install&#8212;rails&#8217; in your rails root, and it will automatically create config/skynet_config.rb and require environment.rb.</p>
<p>Skynet currently supports 2 message queue systems, TupleSpace and Mysql. By default, the TupleSpace queue is used as it is the easiest to set up, though it is less powerful and less scaleable for large installations. If you pass&#8212;mysql to skynet_install, it will assume you are using the mysql as your message queue.</p>
<h2>Starting Skynet</h2>
<p>Once it is installed in your application, you can run skynet from your applications root directory with:</p>
<pre><code>$ skynet start [--workers=N]</code></pre>
<p>This starts a skynet tuple space message queue and 4 workers. You can control how many workers to start per machine
by passing&#8212;workers=N.</p>
<h2>More Info</h2>
<p>For more info check out the <a href="doc/index.html"><span class="caps">README</span></a></p>
<p>Comments are welcome. Send an email to &#8220;Adam Pisoni&#8221; apisoni at</p>
<h2><span class="caps">LICENSE</span>:</h2>
<p>(The <span class="caps">MIT</span> License)</p>
<p>Copyright&#169; 2007 Adam Pisoni,</p>
<p>Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
&#8220;Software&#8221;), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:</p>
<p>The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.</p>
<p><span class="caps">THE SOFTWARE IS PROVIDED</span> &#8220;AS IS&#8221;, <span class="caps">WITHOUT WARRANTY OF ANY KIND</span>,
<span class="caps">EXPRESS OR IMPLIED</span>, INCLUDING <span class="caps">BUT NOT LIMITED TO THE WARRANTIES OF</span>
<span class="caps">MERCHANTABILITY</span>, FITNESS <span class="caps">FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT</span>.
<span class="caps">CLAIM</span>, DAMAGES <span class="caps">OR OTHER LIABILITY</span>, WHETHER <span class="caps">IN AN ACTION OF CONTRACT</span>,
<span class="caps">TORT OR OTHERWISE</span>, ARISING <span class="caps">FROM</span>, OUT <span class="caps">OF OR IN CONNECTION WITH THE</span>
<p class="coda">
Adam Pisoni, 30th May 2008<br>
Theme extended from <a href="">Paul Battley</a>
<!-- insert site tracking codes here, like Google Urchin -->
Jump to Line
Something went wrong with that request. Please try again.