This is a WebCrawler created with Beatufiul Soup and etc ! So we have some dependiceces for setup
$ git clone https://github.com/UtkucanBykl/WebCrawler.git
$ cd WebCrawler
$ pip install -r requirements.txt
And after check for the mongodb status if it is : True else:
$ sudo service mongod start
$ python main.py
$ cd WebCrawler
And visit the http://127.0.0.1:12345/
And crawle what you want.