sunlightlabs / datacatalog-data-gov
watch download tarball
public
Forks1Right
Watchers5Right
Description: Import data.gov datasets to the National Data Catalog
Homepage:
Clone URL: git://github.com/sunlightlabs/datacatalog-data-gov.git
name age
history
message
file .gitignore Thu Oct 01 16:21:00 -0700 2009 rake pull works. to do next: rake push [djsun]
file LICENSE.md Thu Oct 01 16:14:18 -0700 2009 Added license [djsun]
file README.md Thu Oct 08 15:57:59 -0700 2009 New config.yml format, validate with schema_hash [djsun]
file app.rb Loading commit data...
directory config/
directory lib/
file rakefile.rb Wed Oct 07 14:51:22 -0700 2009 require_helpers.rb was a bad idea It clashed i... [djsun]
directory spec/ Wed Oct 07 14:51:22 -0700 2009 require_helpers.rb was a bad idea It clashed i... [djsun]
directory tasks/
README.md

Data.Gov Importer for the National Data Catalog

Installation

Install Needed RubyGems

# If you don't already have gemcutter:
gem install gemcutter
gem tumble

# For development and production:
gem install datacatalog
gem install fastercsv
gem install ruby-unindentable
gem install schema_hash

# For testing:
gem install rspec

Set Up Config Files

  • Create config/config.yml using config/config_example.yml as a template.

Run the Tests

  • Run rake test

Usage

Quick Method

  • rake run does everything in one command

For More Control

  • If you want to run one command at a time...
  • rake pull downloads and parses pages from data.gov
  • rake push uploads the data to the National Data Catalog API

Step By Step

  • If you want to run one command at a time...
  • rake config:verify to test your settings
  • rake pull:directory downloads a directory (a list of pages)
  • rake pull:pages downloads pages listed in the directory
  • rake pull:parse parses the downloaded pages
  • rake push uploads the data to the National Data Catalog API

Join The National Data Catalog Community

Here are some ways to get involved:

You might also like to read our inaugural blog post about the project.