a reliable high-level web crawling & scraping framework for Node.js.
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
lib
test
.gitignore version 1.0.2 Nov 4, 2017
.npmignore
.travis.yml add simple unit test and travis ci Nov 12, 2017
LICENSE Initial commit Nov 4, 2017
README.md version 1.3.6 Dec 7, 2017
index.js
package.json

README.md

Webster

npm version Build Status

Overview

Webster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages. Which is different from other crawling framework is that Webster can scrape the content which rendered by browser client side javascript and ajax request.

Docker quick start

pull the example docker image:

docker pull zhuyingda/webster-demo
docker run -it zhuyingda/webster-demo

in the docker runtime cli:

cd /root/webster_runtime/
node demo_producer.js
node demo_consumer.js

Requirements

  • Node.js 8.x+, redis
  • Works on Linux, Mac OSX

Or you can deploy on Docker.

Install

npm install webster

Documentation

You can see more details from here.

License

GPL-V3

Copyright (c) 2017-present, Yingda (Sugar) Zhu