crowd

Crowd computing in Javascript. Distribute tasks to execute in other computers

Warning: This is a super pre-alpha software. It will only works if you are brave and patient. Watch/Star to follow with the progress

Credits: Planking by Matt Brooks

Project goal

Enable a computer to distribute tasks to execute to other computers. Websites can use computing power of visitors to test performance, scientists can distributedly fold DNA, hackers can distributedly crack passwords (:/).

State

• crowd
  • crowd-express
  • crowd-websockets
• crowd-client (needs options)
  • crowd-rest
  • crowd-webworker
  • crowd-webrtc

Install

$ npm install --save crowdjs

How it works

A server sends a Task to a client that will get the code, evaluate it and execute with the data, once done, return a Result to the server.

See a working example.

// Server
var Crowd = require('crowdjs')
var router = require('crowd-express')

var data = {/* data to send to clients */}
function task (data, next) {
  // This will execute on clients 
}

var crowd = new Crowd(data, task)
  .on('data', function(data) {
    // data coming in real time
    console.log("arriving", data)
  })
  .on('end', function(results) {
    // task completed
    console.log("the end")
  })
  .on('error', function(err) {
    // errors on the way
    console.log("errors", err)
  })


app.use('/task', router(crowd))

The way it will work on the client will be:

var worker = require('crowd-webworker')
var crowd = require('crowd-client')
var rest = require('crowd-rest')

var crowd = new Crowd(opts)
  .use(rest('http://127.0.0.1:1200/task'))
  .pipe(worker())
  .start(opts)

You can avoid Express and talk with the distributed clients by sending them tasks and handling the result:

// Send through HTTP or websockets the task
crowd.toJSON() // {"data": String, "code": String}

// Get the result from the client
crowd.handleResult(resultWeGot)

Other usage

Obviously this is not bound to javascript, to REST and to the browser. Server and clients will be available for WebSockets and WebRTC, and you will be free to implement any of the technology you want.

A bit of history

About 1 year and a half ago (April 2013), me, @rmharrisom and @WilliamMayor met together at the Facebook London Hackathon and we got the 5th Prize with CrowdTaskingJS.

The code was very messy, and a year after, inspired by a more decentralized web from my experience in Mozilla, I decided to clean it and wrap it up into a library that everyone can use.