Permalink
Browse files

Update docs. Rename distributed_find.mapreduce to distributed_find.map

  • Loading branch information...
1 parent 8b8ec53 commit 073c2964c865fa4d629d8e64a03df591cc605b32 phlapjack committed May 31, 2008
View
@@ -27,7 +27,7 @@ Skynet::Manager and Skynet Script Runner
- Added printlog logging method which always prints to the log as [LOG]
- Deprecated Skynet.new to Skynet.start
- Mysql Message Queue Adapter - Make delete_expired_messages much safer.
- - rename ActiveRecord::Base.distributed_find.each to ActiveRecord::Base.distributed_find.mapreduce
+ - rename ActiveRecord::Base.distributed_find.each to ActiveRecord::Base.distributed_find.map
- ActiveRecord::Base.distributed_find - Patch submitted by Lourens Naude (lourens@methodmissing.com) which checks the model for the primary_key name as opposed to assuming it is
'id'
- We don't want to use rails constantize so I've temporarily borrowed the method from ActiveSupport inflector and added it to skynet_ruby_extensions.
View

Large diffs are not rendered by default.

Oops, something went wrong.
@@ -25,7 +25,7 @@ I added a rake task to run the skynet migration
You can also choose to run the command (or other commands) yourself in the skynet console
> ruby script/skynet console
-console> User.distributed_find(:all).each(:migrate_favorites)
+console> User.distributed_find(:all).map(:migrate_favorites)
console> u = User.find(:first)
console> u.user_favorites.size
@@ -14,7 +14,7 @@ namespace :db do
desc "Migrate Favorites"
task :migrate_favorites => :environment do
- User.distributed_find(:all).mapreduce(:migrate_favorites)
+ User.distributed_find(:all).map(:migrate_favorites)
end
end
@@ -180,7 +180,7 @@ def run_job_for_batch(batches,&block)
job.run
end
- def mapreduce(klass_or_method=nil,&block)
+ def map(klass_or_method=nil,&block)
klass_or_method ||= model_class
log = Skynet::Logger.get
@@ -207,7 +207,8 @@ def mapreduce(klass_or_method=nil,&block)
run_job_for_batch(batches)
end
- alias_method :each, :mapreduce
+ alias_method :each, :map
+ alias_method :mapreduce, :map
def model_class
@model_class || self.class.model_class
View
@@ -62,74 +62,70 @@
<h2>What is Skynet</h2>
- <p>Skynet is an open source Ruby implementation of Google&#8217;s Map/Reduce framework, created at Geni.com. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers.</p>
+ <p>Skynet is an open source Ruby implementation of Google&#8217;s MapReduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers. If you&#8217;d like to learn more about MapReduce, see my intro at the bottom of this document.</p>
<p>Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a &#8220;peer recovery&#8221; system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special &#8216;master&#8217; servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.</p>
- <h2>Installing</h2>
+ <p>For more detailed documentation see the following:</p>
- <p>Skynet can be installed via RubyGems:</p>
+ <p>Skynet::Job &#8211; The main interface to Skynet; includes an example of how to use Skynet</p>
- <p><pre class='syntax'><span class="ident">sudo</span> <span class="ident">gem</span> <span class="ident">install</span> <span class="ident">skynet</span></pre></p>
+ <p>Skynet::Config &#8211; Configuration options</p>
- <p>Since Skynet is a distributed system, it requires you have a skynet message queue as well as any number of skynet workers running. Skynet workers need to boot with your code in order to run your jobs. The easiest way is to install a skynet config file with skynet_install.</p>
+ <p>bin/skynet[link:files/bin/skynet.html] &#8211; Starting Skynet</p>
- <p><pre class='syntax'><span class="global">$ </span><span class="ident">skynet_install</span> <span class="punct">[--</span><span class="ident">rails</span><span class="punct">]</span> <span class="punct">[--</span><span class="ident">mysql</span><span class="punct">]</span> <span class="constant">YOUR_APP_DIRECTORY</span></pre></p>
+ <p>bin/skynet_install[link:files/bin/skynet_install.html] &#8211; Installing Skynet into a local project</p>
- <p>skynet_install installs a config/skynet_config.rb file into your application directory.</p>
+ <p>There are also some examples in the examples/ directory included with Skynet.</p>
- <p>If you pass&#8212;rails it will assume it is installing in a rails app. If you pass&#8212;mysql it will assume you are using the mysql as your message queue.</p>
+ <h2>Installation</h2>
- <p>Edit this config file to include the libraries you want skynet workers to know about. If you chose to install into a rails application the skynet_config.rb is already requiring your environment.rb file. Now you can create your own Skynet jobs!</p>
+ <p>Skynet can be installed via RubyGems:</p>
- <h2>Starting Skynet</h2>
+ <pre><code>$ sudo gem install skynet</code></pre>
- <p>Once it is installed in your application, you can run skynet from your applications root directory with:</p>
+ <p>or grab the bleeding edge skynet in svn at
+ $ svn checkout svn+ssh://developername@rubyforge.org/var/svn/skynet
+ $ cd skynet; rake install_gem</p>
- <p><pre class='syntax'><span class="global">$ </span><span class="ident">skynet</span> <span class="ident">start</span></pre></p>
+ <h2>Initial Setup</h2>
- <p>This starts a skynet tuple space message queue and 4 workers. You can control how many workers to start per machine
-by passing&#8212;workers=NUMBER_OF_WORKERS.</p>
+ <p>Skynet works by putting &#8220;tasks&#8221; on a message queue which are picked up by skynet workers. The workers execute tasks and put their results back on the message queue. Skynet workers need to load your code at startup in order to be able to execute your tasks. This loading is handled by installing a skynet config file into your app running skynet_install[link:files/bin/skynet_install.html]. </p>
- <p>You can now run the skynet console to play with skynet a little. See Skynet::ConsoleHelper for commands.
-<pre class='syntax'><span class="global">$ </span><span class="ident">skynet</span> <span class="ident">console</span></pre></p>
+ <pre><code>$ skynet_install <a href="</del>-mysql"><del>-rails</a> APP_ROOT_DIR</code></pre>
- <p>Remember, when you change your code, you must stop/start skynet.</p>
+ <p>This creates a file called skynet_config.rb in <span class="caps">APP</span>_ROOT_DIR/config to which you can add the relevant requires. For example, you might have a rails app and want some of that code to run asynchronously or in a distributed way. Just run &#8216;skynet_install&#8212;rails&#8217; in your rails root, and it will automatically create config/skynet_config.rb and require environment.rb.</p>
- <p>For help try:
-<pre class='syntax'><span class="global">$ </span><span class="ident">skynet</span> <span class="punct">--</span><span class="ident">help</span></pre>
-or
-<pre class='syntax'><span class="global">$ </span><span class="ident">skynet</span> <span class="ident">console</span> <span class="punct">--</span><span class="ident">help</span></pre></p>
+ <p>Skynet currently supports 2 message queue systems, TupleSpace and Mysql. By default, the TupleSpace queue is used as it is the easiest to set up, though it is less powerful and less scaleable for large installations. If you pass&#8212;mysql to skynet_install, it will assume you are using the mysql as your message queue.</p>
- <p>Here are some commands you can run in the skynet console.
-<pre class='syntax'>
- <span class="punct">&gt;</span> <span class="ident">stats</span>
- <span class="punct">&gt;</span> <span class="ident">manager</span><span class="punct">.</span><span class="ident">worker_pids</span>
- <span class="punct">&gt;</span> <span class="punct">[</span><span class="number">1</span><span class="punct">,</span><span class="number">2</span><span class="punct">,</span><span class="number">3</span><span class="punct">,</span><span class="number">1</span><span class="punct">,</span><span class="number">1</span><span class="punct">,</span><span class="number">4</span><span class="punct">].</span><span class="ident">mapreduce</span><span class="punct">(</span><span class="constant">Skynet</span><span class="punct">::</span><span class="constant">MapreduceTest</span><span class="punct">)</span>
-</pre></p>
+ <h2>Starting Skynet</h2>
+
+
+ <p>Once it is installed in your application, you can run skynet from your applications root directory with:</p>
- <p>That last command actually took whatever array you gave it and counted the number of times each element appeared in the array. It&#8217;s not a very useful task, but it shows how easy it is to use.</p>
+ <pre><code>$ skynet start [--workers=N]</code></pre>
- <p>To see what Skynet is doing, you may want to tail the skynet logs being written to your log directory.</p>
+ <p>This starts a skynet tuple space message queue and 4 workers. You can control how many workers to start per machine
+by passing&#8212;workers=N.</p>
<h2>More Info</h2>
View
@@ -19,56 +19,50 @@ h4. Sat Mar 31, 2008 4:25pm - 5:15pm PDT @ Portland Ballroom 255
h2. What is Skynet
-Skynet is an open source Ruby implementation of Google's Map/Reduce framework, created at Geni.com. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers.
+Skynet is an open source Ruby implementation of Google's MapReduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers. If you'd like to learn more about MapReduce, see my intro at the bottom of this document.
Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a "peer recovery" system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special 'master' servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.
-h2. Installing
+For more detailed documentation see the following:
-Skynet can be installed via RubyGems:
+Skynet::Job - The main interface to Skynet; includes an example of how to use Skynet
-<pre syntax="ruby">sudo gem install skynet</pre>
+Skynet::Config - Configuration options
-Since Skynet is a distributed system, it requires you have a skynet message queue as well as any number of skynet workers running. Skynet workers need to boot with your code in order to run your jobs. The easiest way is to install a skynet config file with skynet_install.
+bin/skynet[link:files/bin/skynet.html] - Starting Skynet
-<pre syntax="ruby">$ skynet_install [--rails] [--mysql] YOUR_APP_DIRECTORY</pre>
+bin/skynet_install[link:files/bin/skynet_install.html] - Installing Skynet into a local project
-skynet_install installs a config/skynet_config.rb file into your application directory.
-
-If you pass --rails it will assume it is installing in a rails app. If you pass --mysql it will assume you are using the mysql as your message queue.
+There are also some examples in the examples/ directory included with Skynet.
-Edit this config file to include the libraries you want skynet workers to know about. If you chose to install into a rails application the skynet_config.rb is already requiring your environment.rb file. Now you can create your own Skynet jobs!
+h2. Installation
-h2. Starting Skynet
+Skynet can be installed via RubyGems:
-Once it is installed in your application, you can run skynet from your applications root directory with:
+ $ sudo gem install skynet
-<pre syntax="ruby">$ skynet start</pre>
+or grab the bleeding edge skynet in svn at
+ $ svn checkout svn+ssh://developername@rubyforge.org/var/svn/skynet
+ $ cd skynet; rake install_gem
-This starts a skynet tuple space message queue and 4 workers. You can control how many workers to start per machine
-by passing --workers=NUMBER_OF_WORKERS.
+h2. Initial Setup
-You can now run the skynet console to play with skynet a little. See Skynet::ConsoleHelper for commands.
-<pre syntax="ruby">$ skynet console</pre>
+Skynet works by putting "tasks" on a message queue which are picked up by skynet workers. The workers execute tasks and put their results back on the message queue. Skynet workers need to load your code at startup in order to be able to execute your tasks. This loading is handled by installing a skynet config file into your app running skynet_install[link:files/bin/skynet_install.html].
-Remember, when you change your code, you must stop/start skynet.
+ $ skynet_install [--rails] [--mysql] APP_ROOT_DIR
-For help try:
-<pre syntax="ruby">$ skynet --help</pre>
-or
-<pre syntax="ruby">$ skynet console --help</pre>
+This creates a file called skynet_config.rb in APP_ROOT_DIR/config to which you can add the relevant requires. For example, you might have a rails app and want some of that code to run asynchronously or in a distributed way. Just run 'skynet_install --rails' in your rails root, and it will automatically create config/skynet_config.rb and require environment.rb.
+Skynet currently supports 2 message queue systems, TupleSpace and Mysql. By default, the TupleSpace queue is used as it is the easiest to set up, though it is less powerful and less scaleable for large installations. If you pass --mysql to skynet_install, it will assume you are using the mysql as your message queue.
+
+h2. Starting Skynet
+
+Once it is installed in your application, you can run skynet from your applications root directory with:
-Here are some commands you can run in the skynet console.
-<pre syntax="ruby">
- > stats
- > manager.worker_pids
- > [1,2,3,1,1,4].mapreduce(Skynet::MapreduceTest)
-</pre>
-
-That last command actually took whatever array you gave it and counted the number of times each element appeared in the array. It's not a very useful task, but it shows how easy it is to use.
+ $ skynet start [--workers=N]
-To see what Skynet is doing, you may want to tail the skynet logs being written to your log directory.
+This starts a skynet tuple space message queue and 4 workers. You can control how many workers to start per machine
+by passing --workers=N.
h2. More Info

0 comments on commit 073c296

Please sign in to comment.