PythonWebCrawler

PythonWebCrawler is a python web crawler created to play the wikipedia game Get to Philosophy using Scrapy.

Tech

PythonWebCrawler uses:

• Scrapy - awesome Web Scraping Tool
• Re - used to be able to distingush between the strings and filter between them
• Time - used to avoid heavy load on the website

Installation

PythonWebCrawler requires [Python] to run.

Install the mentioned dependencies then to run just :

$ cd wikiphilo
$ scrapy crawl wikipedia_philosophy -s LOG_ENABLED=False 

Todos

• Write Tests

License

MIT

Free Software, Hell Yeah!