Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Branch: master
Fetching contributors…

Cannot retrieve contributors at this time

35 lines (25 sloc) 0.753 kB

Distributed Web Crawler Sample using connected Workers

A distributed web crawler sample. A central server has queues with links to process. The topology can be manually launched in many nodes. Each node is a worker that can send and receive messages from the other ones.

Install

Execute

npm install

to install the dependencies.

Usage

Launch the server

node server.js <url> [<url>... ]

Example

node server.js http://ajlopez.wordpress.com

Server exposes a queue server using port 3000. It listens to topology workers at port 3001.

Launch one or more clients

node stormapp.js [port]

The node starts to listen other nodes at port. It communicates its address to the topology server listening at port 3001.

Jump to Line
Something went wrong with that request. Please try again.