Skip to content

u-crawler v0.1.6

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Jul 16:27
· 25 commits to main since this release
v0.1.6
5b4617d

Usage

Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because

  1. It is an exe file, which some browsers treat as being suspicious
  2. It is not signed (getting a signing certificate is pretty expensive!)

Features

  • Uses requests, BeautifulSoup and Selenium with headless Chrome
  • Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
  • Respects robots.txt
  • Writes results in structured JSON files per area of interest and overall
  • Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
Assets 5
Loading