Repository navigation
Releases: obobed/u-crawler
Release list
u-crawler v0.3.12
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.3.7
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.3.6
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.3.1
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.2.3
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.2.2
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.1.9
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.1.8
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.1.7
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Also if your browser says it couldn't download the file, use Firefox!
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument
u-crawler v0.1.6
Usage
Just open unzip the folder for your OS and run u-crawler!
You might get a warning on Windows about it being malicious, you can safely ignore it. This happens because
- It is an exe file, which some browsers treat as being suspicious
- It is not signed (getting a signing certificate is pretty expensive!)
Features
- Uses requests, BeautifulSoup and Selenium with headless Chrome
- Scrapes areas of interest (eg. Engineering and Related Technologies) and the undergraduate programs in each one
- Respects robots.txt
- Writes results in structured JSON files per area of interest and overall
- Logs crawl status and errors in logs/ and debugging information if --debug or -d is passed as an argument