Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
JavaScript CoffeeScript Shell
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
app
remote_worker
views
.gitignore
.travis.yml
Procfile
README.md
package.json

README.md

Remote Worker

RemoteWorker is a set of scripts to help you build infinitely scalable web Workers.

Why?

Browsers, Javascript and high speed connections are ubiquitous, and Javascript is finally becoming fast enough to help with your computations (see http://shootout.alioth.debian.org).

As the Shootout shows, Javascript (in Chrome's V8) is currently running between 2x as fast and 100x as slow when compared to native C. At the median, Javascript is 4x as slow as C. So using RemoteWorker makes sense if you think that you'll get 4x more workers working for you in comparison to your native computing capacity or desire to spin up an Amazon EC2 farm.

Goals

  • Make writing Workers simple.
  • Protect browsers from rogue workers.
  • Provide a suite of useful built-ins to make accomplishing common tasks simple.

Features

  • Simple setup
  • Automatic loading of dependent scripts
  • Automatic startup
  • Manage multiple workers
  • Automatic timeout/termination of workers
  • Management and throttling of AJAX requests
  • Utility functions

How to use it

Script Tag

The only modification you typically need to make to your HTML in order to get your remote workers going is to add a script tag:

<script
    name="remote-worker-script"
    type="text/javascript"
    src="remote_worker.js" 
    data-debug="1"
    data-run="0"
    data-dependencies="jquery-1.6.2.min.js,underscore.js"
    data-worker-script="your_workerworker.js"
    data-worker-count="4"
    data-request-delay="2000"
    data-endpoint="/your endpoint">
</script>

See the examples for more detail. Clearly, the 'src' attribute must have the path to remote_worker.js, so set that appropriately. Most of that script tag is standard; however, 5 attributes are custom:

  • data-debug: tells the controller to output log information. Otherwise, @log is a no-op.
  • data-dependencies: these are scripts that the controller will load. Note: don't include jQuery or underscore.js if you're already including them elsewhere in your page.
  • data-worker-script: the script the worker should import.
  • data-worker-count: how many workers to spawn.
  • data-run: tells the controller script to run itself immediately.
  • data-request-delay: tells the controller script how long to aggregate the workers' requests to the server before shipping them to the server. This keeps RemoteWorker from being an annoying bit of software on the client and keeps the server's request load down. This is in milliseconds and defaults to 5000 milliseconds.
  • data-endpoint: tells the controller script where to send requests from the workers.

The Controller

The creates and manages a pool of Workers. It also provides communications mechanisms for the Workers to talk with the outside world.

Though the Controller should be fine for most users, you can subclass the Controller. If you do subclass Controller, then don't use the 'run' attribute on the script because the 'run' attribute will cause the base Controller to run itself before your controller is loaded. You can subclass and startup your controller up by creating a controller script as follows:

# Create my exciting new controller
class MyController extends window.RemoteWorker.Controller
    sample_variable: null

# Keep a reference to the old Controller, then swap in the new hotness
window.RemoteWorker.OriginalController = window.RemoteWorker.Controller
window.RemoteWorker.Controller = MyController

# Instantiate MyController with the 'run' option so that it starts up immediately
window.RemoteWorker.my_controller = new window.RemoteWorker.Controller(run:true)

See the 'life' example for a much longer version.

The Worker

The Worker provides a lot of functionality and the simplest instantiation of a worker.coffee is:

importScripts("../remote_worker.js")
the_worker = new RemoteWorker.Worker()

You'll probably want to subclass Worker in order to do your work. You can do that in CoffeeScript or in Javascript. Please make sure to call super() so that the base class' bits are run. A simple example in CoffeeScript (note: 'self' is the global scope inside of an HTML5 Worker):

class TheWorker extends self.RemoteWorker.Worker
    start: () ->
        @super()
        @postMessage {method:'log', message:"I'm running!"}
        @data = 1
    step: (data) ->
        @super()
        if @state == 'running'
            @postMessage {method:'log', message:"I'm stepping! " + @data.toString()}
            # Do some work
            @data += 1
            # Let the controller know I'm finished
            @postMessage {method:'stepComplete'}
the_worker = new TheWorker()

Note: it's important to call 'stepComplete' in order to tell the controller that the worker has completed its quanta of work. If the worker does not tell the controller that it's completed its step, the controller will treat the worker as a rogue and will terminate it.

Examples

  • skeleton : a very basic implementation. Uses the base classes exclusively.
  • benchmark : runs an N-body simulation (from Debian's Shootout) on multiple workers. Try it.
  • pingpong : the worker sends a 'ping' message to the server and the server sends a 'pong' in return, lather, rinse, repeat.
    The server requires NodeJS. Try it.
  • life : runs eight Games Of Life in the browser, paints eight canvases and reports back to the server every 20 iterations or so. The server requires NodeJS. Try it. See the reports.

The server is running at remoteworker.herokupapp.com and is storing data at MongoHQ.

Rogue Workers

Workers can take excessive processor cycles on users' machines and RemoteWorker tries to prevent that from occuring.
Of course, you can circumvent the restrictions, so RemoteWorker cannot prevent malicious Rogue Workers. That said, RemoteWorker provides tools to help you write workers which do not harm users' web experience.

The way RemoteWorker tries to prevent excessive CPU usage is by terminating Workers that don't complete their "step" within a couple of the controller's "heartbeat" intervals. By default, the controller's heartbeat runs every 100ms and a worker must reply within 2 intervals, so any worker that does not send a stepComplete message within 200ms will be terminated. To help workers understand when to complete a step, they can check this.shouldCompleteStep() and stop/save their work until the next step starts.

TODO

  • Clean up code.. I've smooshed it all together so that it works, but it probably needs a refreshing.
  • Extend features.
  • Increase code documentation
Something went wrong with that request. Please try again.