A distributed web crawler sample. A central server has queues with links to process. The topology can be manually launched in many nodes.
to install the dependencies.
Launch the server
node server.js <url> [<url>... ]
node server.js http://ajlopez.wordpress.com
Launch one or more clients