Raw data available

[rufuspollock]

Establish a raw data s3 bucket with cleaned OS data in it. It has following structure

{dataset-name}/datapackage.json
{dataset-name}/... data files e.g. file1.csv

Questions

bucket name / location?

Propose data.openspending.org

• what is http://data.openspending.org/ - ??
• what is http://archive.openspending.org/ - archive of OS source data files

Nice index

Put in a directory index - https://github.com/rgrp/s3-bucket-listing

How do we get this out of OS atm

Can we do this at the DB level (even just using postgres copy!) (via API is impossible for large datasets - i imagine we can't stream 3gb of data down over the web app ...)

Why

I want to do analysis / queries on OS data that are not supported (or too "costly") by the API - cf #3 (e.g. what are top recipients of uk gov spending ...). To do this I need the raw CSV so I can load into my local postgres / hadoop / bigtable ...

Aside: this in fact could be the import format - these could be the cleaned files we loaded into OS (which would move most of the ETL out of OS core but that's a completely separate discussion ...)

[trickvi]

What would be the use case? Would Amazon Glacier work or is this a short retrieval time service?

[rufuspollock]

This would want live retrieval so glacier is out but reduced redundancy would be fine. But that's a detail.

[rufuspollock]

@pudo does the 25k ETL spit out nice CSVs? Could they be auto-pushed to s3?

[rufuspollock]

OK, http://data.openspending.org has a nice index page :-)

[mk270]

Is this linked from anywhere?

[rufuspollock]

@mk270 not yet ...

[pwalsh]

Closing because:

• data.openspending.org exists
• This whole concept is at the basis of Openspending Next anyway