Skip to content
Quickly download, clean up, and install ecological datasets into a database management system
Python Inno Setup Shell
Latest commit 172a3ba Jun 25, 2016 @henrykironde henrykironde committed on GitHub Merge pull request #539 from ethanwhite/fix-int-error
Remove check for datatype parameter being > 0
Failed to load latest commit information.
docker Add Dockerfile Nov 6, 2014
docs improve regression tests Jun 8, 2016
engines Add bytes check to postgres engine and remove unnecessary code Jun 22, 2016
lib Remove check for datatype parameter being > 0 Jun 25, 2016
scripts Match BBS's change in filename capitalization (again) Jun 24, 2016
test Unicode changes for python3 compatibility Jun 22, 2016
.gitignore Python 3 compatibility changes Jun 7, 2016
.travis.yml Unicode changes for python3 compatibility Jun 22, 2016 Release v1.8.3 Feb 12, 2016
CITATION Adding a CITATION file Sep 2, 2013 Adds link to the adding dataset instructions on the website. Aug 25, 2015
LICENSE Moving license into separate LICENSE file Aug 20, 2013 Make sure that CITATION is added to distributions Jul 4, 2014 Move gitter badge to badges area Jun 8, 2016 Add absolute imports and builtins imports Jun 16, 2016 Revert encoding to avoid decoding issues in Python 2 Jun 24, 2016
appveyor.yml Unicode changes for python3 compatibility Jun 22, 2016 Cleanup and simplify the Linux build script Feb 4, 2014
build_mac don't also remove "python" Feb 5, 2016
build_win Update Windows build script to build installer using Inno Setup Feb 6, 2014 urllib imports Jun 16, 2016
icon.ico replace icon file with a multi-layer .ico file Jan 29, 2016 Add absolute imports and builtins imports Jun 15, 2016 Separating documentation build from deb package build. Jul 26, 2011
modpath.iss Add modpath.iss Jul 11, 2014
osx_icon.icns Add icon to OS X app Jul 6, 2014
requirements.txt Python 3 compatibility changes Jun 7, 2016
retriever_installer.iss Remove unneeded AppVerName directive from Inno Setup script Feb 12, 2016 - write to in standard (non-binary) mode Jun 16, 2016
stdeb.cfg Issue 441: Remove the GUI and all references Jun 2, 2016 Refomat code using PEP 8 standard Feb 18, 2016 urllib imports Jun 15, 2016 Unicode changes for python3 compatibility Jun 22, 2016
version.txt revert verion text after reverting delmoral Jun 15, 2016

Retriever logo

Build Status Build Status (windows) Research software impact Documentation Status License Join the chat at

Large quantities of ecological and environmental data are increasingly available thanks to initiatives sponsoring the collection of large-scale data and efforts to increase the publication of already collected datasets. As a result, progress in ecology is increasingly limited by the speed at which we can organize and analyze data. To help improve ecologists' ability to quickly access and analyze data we have been developing software that designs database structures for ecological datasets and then downloads the data, pre-processes it, and installs it into major database management systems (at the moment we support MySQL, PostgreSQL, SQLite, and Microsoft Access).

Once the EcoData Retriever has loaded the data into the database it is easy to connect to the database using standard tools (e.g., MS Access, Filemaker, etc.).The EcoData Retriever can download and install small datasets in seconds and large datasets in minutes. The program also cleans up known issues with the datasets and automatically restructures them into a format appropriate for standard database management systems. The automation of this process reduces the time for a user to get most large datasets up and running by hours, and in some cases days.

Installing (binaries)

Precompiled binaries the most recent release are available for Windows, OS X, and Ubuntu/Debian at the project website.

Installing From Source

To install the EcoData Retriever from source, you'll need Python 2.6+ with the following packages installed:

  • xlrd

The following packages are optional

  • PyMySQL (for MySQL)
  • sqlite3 (for SQLite)
  • psycopg2 (for PostgreSQL)
  • pyodbc (for MS Access - this option is only available on Windows)

To install from source

  1. Clone the repository
  2. From the directory containing, run the following command: python install. You may need to include sudo at the beginning of the command depending on your system (i.e., sudo python install).
  3. After installing, type retriever from a command prompt to launch the EcoData Retriever

Using the Command Line

After installing, run retriever update to download all of the available dataset scripts. To see the full list of command line options and datasets run retriever --help. The output will look like this:

usage: retriever [-h] [-v] [-q] {install,update,new,ls,citation,help} ...

positional arguments:
                        sub-command help
    install             download and install dataset
    update              download updated versions of scripts
    new                 create a new sample retriever script
    ls                  display a list all available dataset scripts
    citation            view citation

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -q, --quiet           suppress command-line output

To install datasets, use retriever install:

usage: retriever install [-h] [--compile] [--debug]
                         {mysql,postgres,sqlite,msaccess,csv} ...

positional arguments:
                        engine-specific help
    mysql               MySQL
    postgres            PostgreSQL
    sqlite              SQLite
    msaccess            Microsoft Access
    csv                 CSV

optional arguments:
  -h, --help            show this help message and exit
  --compile             force re-compile of script before downloading
  --debug               run in debug mode


These examples are using Breeding Bird Survey data (BBS) 

Using Install

  retriever install -h   (gives install options)

Using specific database engine, retriever install {Engine}

  retriever install mysql -h     (gives install mysql options)
  retriever install mysql --user myuser --password ******** --host localhost --port 8888 --database_name testdbase BBS

install data into an sqlite database named mydatabase.db you would use:

  retriever install sqlite BBS -f mydatabase.db

Using download

  retriever download -h    (gives you help options)
  retriever download BBS"
  retriever download BBS --path C:\Users\Documents

Using citation
  retriever citation   (citation of the retriever engine)
  retriever citation BBS   (citation of BBS data) 


Development of this software was funded by the Gordon and Betty Moore Foundation's Data-Driven Discovery Initiative through Grant GBMF4563 to Ethan White and the National Science Foundation as part of a CAREER award to Ethan White.

Something went wrong with that request. Please try again.