The task is to create a Web Crawler that can crawl inforamtions and index web pages when a set a keywords is provided to it. It will be a good bot. It will mostly crawl through news and events.
.gitignore
.\crawlenv
scrapy 2.5.1;
python 3.6.5;
virtualenv (20.4.3);
beautifulsoup4 4.10.0;
pip 2.3.1
Flask 2.0.2
How to start the project:
- From the root directory, inside the shell, type .\crawlenv\Scripts\activate
- Next cd to Scraper
- Run python main_generic.py
- Paste a link to scrape