Skip to content
a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
test_celery Delete tasks.pyc May 20, 2018
README.md
docker-compose.yml
dockerfile
requirements.txt

README.md

scaleable-crawler-with-docker-cluster

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine Check out here for full document

You can’t perform that action at this time.