Skip to content
middleware based lightweight crawler framework
Branch: master
Clone or download
Latest commit cb70ce2 Aug 19, 2016
Type Name Latest commit message Commit time
Failed to load latest commit information.
src Add caching function Aug 19, 2016
.babelrc Initialize Aug 18, 2016
.eslintrc.json Initialize Aug 18, 2016
.gitignore Initialize Aug 18, 2016
.npmrc Initialize Aug 18, 2016
.travis.yml Bye v5 Aug 19, 2016
LICENSE Initial commit Aug 11, 2016
package.json v0.0.3 Aug 19, 2016

stackable-crawler Build Status

middleware based lightweight crawler framework.


  • Customizable pre-request, pre-process middleware stacks (it enables to log, cache, normalize, etc...)
  • Cancelable crawler
  • Customizable caching strategy
  • Parallel crawling
  • Pause/Resume crawling (it enables to sleep crawler)
  • Error handling (it enables to retry)


$ npm i -S stackable-crawler


import StackableCrawler, {
} from 'stackable-crawler';

const crawler = new StackableCrawler({
  prerequest: [
    options => {
      console.log('options:', options);
      return options;
  processor([response, body]) {
    return new Promise((resolve, reject) => {
      saveFileFunction(body, error => {
        if (error) return reject(error);

crawler.on('error', (error, url) => {
  console.error(error, url);


What can I do in prerequest?

prerequest middlewares stack can have sideeffect about requesting options. options is request module's request option. If prerequest middleware throw CancelRequest error, to request to that url was canceled.

What can I do in preprocess?

preprocess middlewares stack can have sideeffect about response, body, requestOptions. They are also from request.

Friendly crawler?

Use sleep function

const sleepCrawler = (crawler, sleepTime, interval) => {
  setTimeout(() => {
    crawler.once('stopped', () => {
      setTimeout(() => crawler.start(), sleepTime);
  }, interval);


StackableCrawler class (Default export)

Class of crawler. Extends EventEmitter3.


Take one argument, configure object.

  concurrency: 1, // max # of parallel crawling
  prerequest: [], // prerequest middleware functions
  requestCache() {}, // cache strategy
  preprocess: [], // preprocess middleware functions
  processor() {}, // main callback to handle crawled Document
prerequest middleware

Function that process requestOptions. Default argument is { uri: url }. Function must return new (mutated) requestOptions or Promise[requestOptions].

requestCache function

Function that returns cached value or undefined. Type of cached value is T or Promise[T]. If cached value returned, crawler pass through that values to processor function. If undefined returned, crawler fetch document as usual.

preprocess middleware

Function that process [response, body, requestOptions]. response and body are fetching result of request module. It also return new argument or Promise.

processor function

Main function. You can do everything here. It can return promise.


Add url to crawling task queue.


Pause crawler. If crawler has one more running tasks, these are still running until finished, but no more run new task.


Resume crawler if paused.

crawler#on(), crawler#once(), ...

These methods are inherit from EventEmitter3.

available event and args

  • event: error, args: [error, url]
  • event: stopped

CancelRequest class (Named export)

Error class. If crawler throws CancelRequest error throw new CancelRequest(), crawler stop to request that url with no error.

middlewares object (Named export)

Bundled simple middlewares.


Filter only valid url. If url is invalid, this middleware throw CancelRequest error.


Function that returns simple in memory cache middleware. Do urlCache() when use. If requesting url was already cached, this middleware throw CancelRequest error.


Very simple console.log middleware


Replace body to cheerio object ($). Next middlewares and processor call with [response, $, requestOptions]



You can’t perform that action at this time.