Webster is A Powerful and Extensible Web Crawling Framework for Node.js application. You can use Webster to crawl websites and extract structured data from their pages.
Docker quick start
pull the example docker image:
docker pull zhuyingda/webster-demo docker run -it zhuyingda/webster-demo
node demo_producer.js env MOD=debug node demo_consumer.js
- Node.js 10.x+, redis
- Works on Linux, Mac OSX
Or you can deploy on Docker.
npm install webster
Usage on Raspbian Platform
sudo apt install chromium-browser chromium-codecs-ffmpeg env MOD=debug EXE_PATH=/usr/bin/chromium-browser node demo_consumer.js
You can see more details from here.
This project exists thanks to all the people who contribute. [Contribute].
Become a financial contributor and help us sustain our community. [Contribute]
Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]
Copyright (c) 2017-present, Yingda (Sugar) Zhu