Web indexer and scraper runner (MIT)
JavaScript Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


build status

spyder - Indexer and scraper runner

spyder provides basic architecture for running indexers and scrapers. It comes as a CLI tool which you then provide configuration using either -c or --config parameter (spyder -c ./config.js).

It is also possible to provide a directory with spyder_config.js in it. In that case spyder tries to load the configuration automatically. Example: spyder demo.

You can also pass additional parameters to both commands. They will override the default configuration.

As a Module

spyder can also be used as a regular Node module. It expects configuration as its parameter like this:

var spyder = require('spyder');



Consider the following config.js for basic configuration:

module.exports = {
    // workers
    initializer: require('./init'), // optional
    indexer: require('./indexer'),
    scraper: require('./scraper'),

    // events
    onError: require('./error'),
    onResult: require('./result'),
    onFinish: require('./finish'),

    // other
    variance: 5000 // variance between scrape operations in ms


spyder provides three workers into which you may attach actual functionality. initializer is executed once when spyder process is started. You may set auth keys and such there. indexer is run once per scraping round. scraper is executed per each url returned by indexer.


initializer is optional. A basic implementation could look like this:

module.exports = function(o, cb) {
    // do something with o now
    // ...

    cb(); // done

The first parameter will contain arguments passed to spyder process. This behavior is the same for all workers.


An indexer could look like this:

module.exports = function(o, cb) {
    // index some page or pages here

    // once finished, cb
    cb(null, [
        'http:// ...' // url to target to scrape

Remember to return the urls you want to scrape here. In case you run into error, pass it as the first parameter to the callback.


A scraper could look like this:

module.exports = function(o, url, cb) {
    // scrape the content from url now

    // once finished, cb
    cb(null, {
        title: 'Demo'

The same idea as earlier applies here. First the function receives arguments passed to spyder, then url to scrape and finally a callback to call when finished.


In case an error is received, module defined at onError is defined. When a scraping result is received, onResult module is invoked. Once the whole process has finished, onFinished is invoked. Like above each handler receives arguments. You can for instance inject an object there at initializer and then use that to perform some operation. To give you an idea of what these files should look like, consider the following.


module.exports = function(o, err) {
    // let's just log errors for now
    // this is also the default behavior. if you don't provide a handler,
    // spyder defaults to this


module.exports = function(o, result, cb) {
    // got some scraping result now, do something with it
    // spyder defaults to console.log (handy during development)

    // the callback is optional and allows you to communicate possible errors
    cb(new Error('Demo error'));


module.exports = function(o) {
    // spyder default


  • variance - Use variance to add arbitrary, random delay between scrape operations to make traffic look more irregular.


spyder is available under MIT. See LICENSE for more details.