This is a basic web crawler program. You can read more about this program on the following link, I have explained how it is written and used --
$ git clone https://github.com/An4ndita/Basic-web-crawler
$ cd Basic-web-crawler
$ python crawler.py
Enter the domain to crawl >>