A simple to use Promise-based web scraper with local caching.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib
test
.eslintignore
.eslintrc
.gitignore
.npmignore
.travis.yml
CHANGELOG.md
CONTRIBUTING.md
DOCUMENTATION.md
LICENSE
README.md
index.js
package.json

README.md

raspar

A simple to use Promise-based web scraper with local caching.

Build Status codecov Dependency Status Known Vulnerabilities bitHound Overall Score NPM Version Greenkeeper badge Latest Documentation

Usage

const raspar = require('raspar');

raspar.fetch('http://www.google.com/humans.txt').then(res => {

    console.log(res.body);

});
const raspar = require('raspar');

raspar.fetch(['http://www.google.com/humans.txt']).then(res => {

    console.log(res[0].body);

});

Options

const raspar = require('raspar');

const options = {
    'cacheDirectory': 'temp/cache/',
    'cacheMemory': false,
    'requestOptions': {
        'headers': {
            'User-Agent': 'request'
        },
        'method': 'POST'
    },
    'ttl': 1800
};

raspar.fetch('http://www.google.com/humans.txt', options).then(res => { });
Name Description Default Value
cacheDirectory Directory to store cache. temp/cache/
cacheMemory Store cache in memory instead of local files. false
ttl TTL (Time to live) in seconds for local/memory based cache. 1800

Documentation

View full documentation here.