Documentation to support upcoming data platform API and data sets
CSS HTML Shell JavaScript Clojure
Latest commit 265134e Sep 29, 2016 @sleitner sleitner committed on GitHub HMDA description tweak (#59)
Temporarily remove the bulletin article year and fix spelling error
Permalink
Failed to load latest commit information.
resources HMDA description tweak (#59) Sep 29, 2016
src/cfpb/api Upped required version of Qu Mar 5, 2014
.gitignore Update qu-core dependency to 1.1.3-SNAPSHOT Jul 18, 2014
README.md
project.clj Update qu-core dependency to 1.1.3-SNAPSHOT Jul 18, 2014
sample_config.edn Added sample configuration Apr 25, 2014

README.md

CFPB Public Data API

This is the source project for the CFPB public data API at http://api.consumerfinance.gov. It contains up-to-date load scripts for populating data in that API. Note, however, that the resources/static/ directory is a work in progress; refer to https://github.com/cfpb/qu/tree/master/resources/static for the current static resources.

Create a config file

cp sample_config.edn config.edn

Edit that file with appropriate values, including MongoDB connection information.

Loading data

To load the HMDA dataset, run lein repl and enter the following:

(-main "config.edn") ;; use the name of your config file. config.edn is an example
(require 'qu.loader)
(in-ns 'qu.loader)
(load-dataset "hmda")

Then close your REPL, using Ctrl-D. This does start the web server, so you may want to use a different configuration that starts it on an unpublished port.

Speeding up data load

You can take advantage of more CPU and Ram by concurrently loading chunks of data by splitting the large data file into smaller files with 1 million records each.

./hmda_split_csv.sh hmda_lar_all_2012.csv split_hmda_2012_

Then update the definition.json file to load these split files instead of the single CSV. Edit the hmda_lar sources section and replace it with a list of all the split files for the year you're targeting.

For example, for 2012, it looks like this:

   "sources": [
"split_hmda_2012_aa",
"split_hmda_2012_ab",
"split_hmda_2012_ac",
"split_hmda_2012_ad",
"split_hmda_2012_ae"
]