a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
test_celery
README.md
docker-compose.yml
dockerfile
requirements.txt

README.md

scaleable-crawler-with-docker-cluster

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine Check out here for full document