A prototype of an ETL and API for Open Denton
This is a prototype of an application that imports data from the city of Denton's open data project into a permanent schema, and offers a flexible API on top of that schema. The db is PostgreSQL and the framework for the API is Sinatra.
Install & bootstrap
- Install PostgreSQL
- Make sure you have ruby installed
- From the project directory,
gem install bundler
rake db:create db:schema:load
- To run the server locally:
bundle exec ruby app/app.rb
- To import all available data, go to
- To delete all current data, go to
- See example APIs locally (the 'year' params can vary between 2008 and 2014):
If you get the following error:
➜ psql: FATAL: database "<user>" does not exist
Run the shell command,
createdb. This creates a database with the name of the current user.
Source: Stack Overflow Article.
If you used Brew to install Postgres, you can also use services
to start and stop the
psql background service.
➜ brew tap homebrew/services ## if you don't have it already installed ➜ brew services start postgresql ➜ brew services start postgresql
If the app crashes with the following warning:
$ ActiveRecord::ConnectionTimeoutError - could not obtain a connection from the pool within 5.000 seconds (waited 5.001 seconds); all pooled connections were in use:
You broke it!
Just kidding.. but the app will need to be restarted. We still need to build in some logic for handling blocking calls. Thanks fer yor patience!
- Browse http://data.cityofdenton.com/dataset
- Find Dataset, click on CSV
- Select CSV, Explore, Preview
Data APIand find "Query Example (first 5 results)", grab
For examples below, the Resource ID is
5980ea98-ce00-4f7d-9ee3-114006f78f59 and the DB table will be
- in terminal open
- Add Dataset Endpoint
- Add new model in
- Add new migration in
- Commit on feature branch