Migrate data from current ckan instance to new DB (spreadsheet) - Aug 2014

[rufuspollock]

• download and cache current CKAN data
• transform to new structure
• incorporate existing geocoding work
• convert to CSV
• import into spreadsheet

[rufuspollock]

CKAN exported data looks like:

{
  "maintainer": "",
  "name": "a2gov_org",
  "metadata_modified": "2012-12-18T06:24:02.918Z",
  "author": "City of Ann Arbor",
  "author_email": "",
  "title": "Ann Arbor, Michigan",
  "notes": "City of Ann Arbor's Open Data Catalog (USA)",
  "entity_type": "package",
  "site_id": "datacatalogs.org",
  "maintainer_email": "",
  "url": "http://www.a2gov.org/data/Pages/default.aspx",
  "state": "active",
  "tags": [
    "ctic"
  ],
  "groups": [
    "unitedstates"
  ],
  "license_id": "",
  "revision_id": "f2ba3555-d5bc-42c1-a4f1-45f98fdc34fc",
  "indexed_ts": "2013-03-11T20:05:24.13Z",
  "index_id": "28532c09512d3c7b0b21073920744d05",
  "id": "ec73f274-6986-4f46-921f-07ac7c7f4f65",
  "metadata_created": "2011-06-27T18:12:57.439Z",
  "extras": {
    "language": "en",
    "spatial_text": "Ann Arbor, Michigan",
    "spatial": "us"
  }
},
{
  "name": "africa_open_data",
  "metadata_modified": "2013-03-15T07:17:26.251Z",
  "title": "Africa Open Data",
  "url": "http://africaopendata.org/",
  "notes": "Africa's largest central repository for Government, Civil Society, Corporate and Donor Agency Data.",
  "author": "Africa Open Data",
  "site_id": "datacatalogs.org",
  "indexed_ts": "2013-06-10T14:36:29.556Z",
  "state": "active",
  "extras": {
    "language": "en",
    "spatial_text": "Africa",
    "spatial": "Africa"
  },
  "groups": [
    "ckan",
    "africa"
  ],
  "id": "b061d0cc-6d8a-4c9e-8dd5-752505304efe",
  "license_id": "",
  "revision_id": "52ffa1d5-5224-4a89-84e1-bcf25ca13e98",
  "index_id": "6c1caa1a05a976ce801be5c57bb8f3db",
  "entity_type": "package",
  "metadata_created": "2013-03-15T07:17:26.251Z"
},

[rufuspollock]

Ok most done. Won't do geolocation on the new stuff (just pulled geolocation from existing data).