Distributed web-crawler based on Docker

Introdution

This is a small web-crawler project used distributed architecture and based on Docker. It also used cool OpenSource projects :

It's mainly built on Python Language, but you can use many others Languages (C, C++, Java, PHP, NodeJS, ...) to implement.

Step 1 : Build docker image

git clone https://github.com/khanhicetea/distributed-webcrawler.git
cd distributed-webcrawler
docker-compose build

Step 2 : Run init containers

docker-compose up -d

Step 3 : Run crawlers

docker exec -it distributedwebcrawler_worker1_1 /entrypoint.sh crawler [number_of_crawlers]

Licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
Latest commit History 21 Commits
bloomd_config		bloomd_config
rethinkdb_data		rethinkdb_data
supervisord_config		supervisord_config
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
crawler.py		crawler.py
dequeue_worker.py		dequeue_worker.py
docker-compose.yml		docker-compose.yml
enqueue_worker.py		enqueue_worker.py
entrypoint.sh		entrypoint.sh
main.py		main.py
parameters.yml		parameters.yml
requirements.txt		requirements.txt