A logger plugin for sandcrawler.
JavaScript HTML
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
test
.gitignore
.npmignore
README.md
defaults.json
index.js
package.json

README.md

sandcrawler-logger

Simple logger to plug into one of your sandcrawler spiders for immediate feedback.

Installation

You can install sandcrawler-logger through npm:

npm install sandcrawler-logger

Usage

var sandcrawler = require('sandcrawler'),
    logger = require('sandcrawler-logger');

var spider = sandcrawler.spider('MyFancySpider')
  .use(logger())
  .url('http://nicesite.org')
  .scraper(function($, done) {
    done(null, $('title').text());
  })
  .run();

Options

  • color ?string ['magenta']: color attributed to your spider. This is useful when running several spiders at once when you want to clearly distinguish their respective logs.
  • level ?string ['debug']: Threshold level for the log, order being: debug, verbose, info, warn and error.
  • pageLog ?boolean [true]: in case of a phantom spider, should we display the web page's log and errors?

Example

var sandcrawler = require('sandcrawler'),
    logger = require('sandcrawler-logger');

var spider = sandcrawler.spider('MyFancySpider')
  .use(logger({color: 'blue', level: 'warn'}));

Spider's logger

When plugging the logger onto your spider, this one will be attached a logger property so you can push custom information through the means of the plugin if needed.

var sandcrawler = require('sandcrawler'),
    logger = require('sandcrawler-logger');

var spider = sandcrawler.spider('MyFancySpider')
  .use(logger())
  (...)
  .result(function(err, req, res) {
    this.logger.info('Page title: ' + res.data);
  })
  .run();

Note that, under the hood, this plugin uses the winston library, on whose documentation you can find more information about how to use the spider.logger property.

License

MIT