This repository has been archived by the owner on Mar 18, 2024. It is now read-only.

Blog Spider

Run

docker-compose up --build -d

Cron Job

run spiders

curl -X POST -H "Content-Type: application/json" -d '{"spiders": ["all"]}' http://127.0.0.1:8080/submit/crawl

backup mongo

docker-compose exec mongodb mongodump -u scrapy -p scrapy -d scrapy -c spider --gzip --archive=/dump/spider.gz
docker-compose exec mongodb mongodump -u scrapy -p scrapy -d scrapy -c category --gzip --archive=/dump/category.gz

React JS Build

docker container run --rm -v `pwd`/www/src/static/:/static/ -w /static/app/ node ./build.sh