Running Crawler on Docker Container

This repository for running crawler from target_url to getting data in database on docker container.

Running on docker:

1. Install and run the Docker machine.

2. Clone this repository

3. Inside the repository:

   • check existing or create new target_url_crawler and data_cralwer as per your requirements.

     NOTE: You can access mysql after running docker on:
     http://<docker-public ip>:8080?server=172.30.0.5)
     For docker public ip, check: docker-machine ip default
     On mac: http://localhost:8080?server=172.30.0.5

4. Change go_spider.py cmd as per your crawler

5. Build and Run the Crawler

    docker-compose up --build
   

Note:

1. Any changes in the service, make sure you first delete respective service conatiner.

2. Can access the volume after running container:

    docker exec -it <container-id/name> bash