v0.1.0

First release!

Usage

Just open a command prompt where you downloaded the exe and run u-crawler.exe!

Features

• Uses requests, BeautifulSoup and Selenium with headless Chrome
• Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
• Respects robots.txt
• Writes results in structured JSON files per area of interest and overall
• Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument