We built Countdera on top of Firebase and used two node servers (written in coffeescript) to manage jobs and nodes in the cluster.
We wrote all of this code for HackTX 2014 in 24 hours where we made it to the finals but didn't place.
The idea behind Countdera was to allow webpages to serve an iframe that would use a viewer's CPU cycles (after they opt-in) to run MapReduce jobs and generate revenue for the hosting website. Researchers or companies would then come to our site and pay to use our massive cluster of these worker nodes to run their computations. This system would have the potential to replace online advertising and was inspired by Tidbit.
The protocol we built Countdera on was inspired by MapReduce but was our own design. It makes some simplifying assumptions but is generally fully-featured. You can read more about the design of the protocol that this repo implements on this website.
Running this as is for yourself
- Create a Firebase and put the name of your Firebase in
- Create the directory
/countdera/outputon the machine you're running the server on.
npm installin the root of the project.
npm startto start the job tracking server and the static webserver on
node bin/ioserver/ioserver.jsto start the server that creates the output files
http://localhost:3000and select to be a worker (a node in the cluster) or make a MapReduce job.
http://your_domain/workerin an embedded iframe for the workers.
Note: Your input files but be available cross domain. This means when you server them you must send the proper CORS headers.
Things that would need to be done to make this production worthy:
- Batching map-output messages. Right now each emission from a mapper is sent to every reducer. For large input files, this would require many millions of messages.
- Possibly using WebRTC instead of Firebase FIFO queues for messages (We didn't do this because most of the WebRTC libraries suck).
- Firebase permissions. Currently people could do bad things to the cluster if they knew the Firebase name.