Parallel Processing for the Rest of Us
Fetching latest commit…
Cannot retrieve the latest commit at this time
_ _ ( ` )_ ( ) `) (_ (_ . _) _) _ ( ) _ . ( ` ) . ) ( _ )_ (_, _( ,_)_) (_ _(_ ,) _ _ ___ _ _ ___ _ ( ` )_ / __| |___ _ _ __| |/ __|_ _ _____ __ ____| | ( ) `) | (__| / _ \ || / _` | (__| '_/ _ \ V V / _` | (_ (_ . _) _) \___|_\___/\_,_\__,_|\___|_| \___/\_/\_/\__,_| _ ( ) _, _ . ( ` ) . ) ( ( _ )_ (_, _( ,_)_) (_(_ _(_ ,) ~ CloudCrowd ~ * A batch-processing system, map-reduce style * Write your scripts in Ruby * Built for Amazon EC2 and S3 * split -> process -> merge * As easy as `gem install cloud-crowd` ~ Getting started ~ # Install the gem (documentcloud-cloud-crowd until the first official release). >> sudo gem install cloud-crowd # Install the CloudCrowd configuration files to a location of your choosing. >> crowd install /etc/cloud-crowd # Now, you can use the full complement of `crowd` commands from inside of # this configuration directory. To see the available commands: >> crowd --help # Edit the configuration files to your satisfaction, and add AWS credentials. >> mate /etc/cloud-crowd/config.yml >> mate /etc/cloud-cloud/database.yml # Write your actions, and install them into the 'actions' subdirectory. # CloudCrowd comes with some default actions as an example. # To spin up the central server (make sure that you include its location # in config.yml), either: >> crowd server # or: >> thin -R config.ru --servers 3 -e production start # Any server that supports Rack should work with the rackup file. # Then, to spin up 10 workers: >> crowd workers start -n 10 # To spin up workers remotely, install the 'cloud-crowd' gem, and copy over # your configuration directory.