Illinois EPA Discharge Monitoring Reports download and import
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore initial commit Oct 9, 2015
README.md Update README.md Mar 15, 2017
create.sql
dmr.py
import.sh initial commit Oct 9, 2015

README.md

il-dmr

This repository contains a couple of scripts for downloading the Illinois EPA Discharge Monitoring Reports (DMR) and importing them into a database.

The main script is dmr.py which downloads the csv files from the EPA website. Some useful examples:

To download all dmr csv files (to il-dmr/csv directory) use

python dmr.py

To download and import csv files (serially, which is good in case of failure) first run create.csv and then

python dmr.py --psql

The import is done using psql -c '\copy so place the relevant PostgreSQL credentials in environment variables. To download a particular year

python dmr.py --year 2015

To download a particular NPDES id (all years, unless --year is specified)

python dmr.py --npdes IL0000035

By default existing csv files will not be re-downloaded. To re-download (useful for updating a year in progress)

python dmr.py -o --year 2015

There are two additional scripts:

create.sql for creating the target dmr table

import.sh for bulk import of csvs after they have been downloaded.