documentcloud / cloud-crowd
watch download tarball
public
Forks14Right
Watchers396Right
Description: Parallel Processing for the Rest of Us
Homepage: http://wiki.github.com/documentcloud/cloud-crowd
Clone URL: git://github.com/documentcloud/cloud-crowd.git
name age
history
message
file .gitignore Thu Sep 17 07:50:59 -0700 2009 moving the default test database to sqlite3, ad... [jashkenas]
file .yardopts Tue Sep 15 05:46:03 -0700 2009 keeping yardopts for yard generation [jashkenas]
file EPIGRAPHS Thu Aug 27 06:51:38 -0700 2009 removing some old examples [jashkenas]
file LICENSE Thu Aug 27 06:30:29 -0700 2009 back to the MIT license, because that's what I ... [jashkenas]
file README Wed Sep 16 08:37:10 -0700 2009 an exorcism of tabs [jashkenas]
file Rakefile Sun Sep 20 08:14:34 -0700 2009 adding --local to 'rake gem:install speeds thin... [jashkenas]
file TODO Mon Oct 19 06:40:04 -0700 2009 clearing todo list [jashkenas]
directory actions/ Tue Nov 03 12:23:14 -0800 2009 removing a now-unneeded to_json [jashkenas]
directory bin/ Sun Aug 23 15:32:54 -0700 2009 moving things around in an attempt to get the r... [jashkenas]
file cloud-crowd.gemspec Thu Nov 19 09:23:55 -0800 2009 cloud-crowd 0.3.1, with better FilesystemStore ... [jashkenas]
directory config/ Fri Nov 06 21:56:43 -0800 2009 added pragdave's patch for Ruby 1.9 compatibili... [jashkenas]
directory examples/ Wed Sep 16 05:13:39 -0700 2009 ready to merge back to master... [jashkenas]
directory lib/ Thu Nov 19 09:23:55 -0800 2009 cloud-crowd 0.3.1, with better FilesystemStore ... [jashkenas]
directory public/ Tue Sep 22 18:06:06 -0700 2009 making sure that long action names don't overfl... [jashkenas]
directory test/ Fri Nov 06 21:56:43 -0800 2009 added pragdave's patch for Ruby 1.9 compatibili... [jashkenas]
directory views/ Tue Sep 22 18:06:06 -0700 2009 making sure that long action names don't overfl... [jashkenas]
directory wiki/ Thu Nov 19 09:23:55 -0800 2009 cloud-crowd 0.3.1, with better FilesystemStore ... [jashkenas]
README
=                                                                               
           _  _                                                                
          ( `   )_                                                             
         (    )    `)                                                          
       (_   (_ .  _) _)                                                        
                                      _                                        
                                     (  )                                      
      _ .                         ( `  ) . )                                   
    (  _ )_                      (_, _(  ,_)_)                                 
  (_  _(_ ,)                                                                   
                                                                               
           _  _               ___ _             _  ___                   _     
          ( `   )_           / __| |___ _  _ __| |/ __|_ _ _____ __ ____| |    
         (    )    `)       | (__| / _ \ || / _` | (__| '_/ _ \ V  V / _` |    
       (_   (_ .  _) _)      \___|_\___/\_,_\__,_|\___|_| \___/\_/\_/\__,_|    
                                                                               
                                                     _                         
                                                    (  )                       
                  _, _ .                         ( `  ) . )                    
                 ( (  _ )_                      (_, _(  ,_)_)                  
               (_(_  _(_ ,)                                                    
                                                                               
                                                                               
                                                                               
  ~ CloudCrowd ~

    * Parallel processing for the rest of us
    * Write your scripts in Ruby
    * Works with Amazon EC2 and S3
    * split -> process -> merge
    * As easy as `gem install cloud-crowd`

    Well-suited for:
    
    * Generating or resizing images.
    * Encoding video.
    * Running text extraction or OCR on PDFs.
    * Migrating a large file set or database.
    * Web scraping.
    
    
  ~ Documentation ~
  
    Wiki: http://wiki.github.com/documentcloud/cloud-crowd
    Rdoc: http://rdoc.info/projects/documentcloud/cloud-crowd
  
  
  ~ Getting started ~
  
    # Install the gem.
    
      >> sudo gem install cloud-crowd
    
    # Install the CloudCrowd configuration files to a location of your choosing.
    
      >> crowd install ~/config/cloud-crowd
    
    # Now, you can use the full complement of `crowd` commands from inside of
    # this configuration directory. To see the available commands:
    
      >> crowd --help
    
    # Edit the configuration files to your satisfaction, add AWS credentials, 
    # and then load the CloudCrowd schema into your configured database.
    
      >> cd ~/config/cloud-crowd
      >> mate config.yml
      >> mate database.yml
      >> [create the database you just configured...]
      >> crowd load_schema
    
    # Write your actions, and install them into the 'actions' subdirectory.
    # CloudCrowd comes with a few default actions as an example.
    
    # To launch the central server (make sure that you include its location
    # in config.yml):
    
      >> crowd server
    
    # The configuration folder also includes 'config.ru', which can be used by
     # any Rack-compliant webserver to run your central server.
    
    # Then, to launch a node of workers:
    
      >> crowd node
    
    # To spin up remote nodes, install the 'cloud-crowd' gem and copy over
    # your configuration directory. Run `crowd node`, and the remote machines
    # will register with the central server, becoming available for processing.
    
    # At this point you can visit your Operations Center at localhost:9173 to 
    # view all of your nodes, ready for action.