scraping millions of records from a public government website
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Flask App
Checker.py
Extractor.py
Keymaker.py
Main.py
Process.py
README.md
Uploader.py
logs.txt

README.md

Web Scraping Sample

A script and tools to scrape a whole country's population data from a web service and put them in a database.
Threaded to make the ~19 million records faster to download.
Separate script to verify downloaded records and a small Flask app to view the data.