New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[super] CKAN + Tabular Data Packages (Import and Export) #126

Closed
rufuspollock opened this Issue Sep 11, 2014 · 10 comments

Comments

Projects
None yet
4 participants
@rufuspollock
Copy link
Contributor

rufuspollock commented Sep 11, 2014

Push (Tabular) Data Packages to CKAN and their data to CKAN DataStore, and get Data Packages from CKAN.

Note: heavily updated May 2016.

Pushing data packages to CKAN and, specifically, data to the DataStore will mean Data Packages can quickly get a nice Data API via the DataStore API

  • Import Tabular Data Packages to CKAN #237
  • Export Tabular Data Packages from CKAN (download CKAN datasets onto disk as a Tabular Data Package) #236

Extras

  • Push "data" to CKAN - not yet data packaged ... (modelled)

General Architecture

Import - #126

Worth emphasizing overlap with #238 (github => ckan) in terms of the import hook / import step.

Source: https://docs.google.com/drawings/d/1XqovADufTR5OYRZW0WjeltcYVyCPZ6OVHyfjLSvuJKg/edit

Existing work

@rufuspollock rufuspollock changed the title Push (Tabular) Data Packages to CKAN (and data to CKAN DataStore) Push (Tabular) Data Packages to CKAN (and data to CKAN DataStore) and Pull (Install) from CKAN Sep 11, 2014

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented Apr 1, 2015

Pushing Data Packages was implemented in dpm last year: http://okfnlabs.org/blog/2014/09/11/data-api-for-data-packages-with-dpm-and-ckan.html

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented Apr 1, 2015

See also https://github.com/rgrp/ckan-import - more of generic import.

@danfowler

This comment has been minimized.

Copy link
Contributor

danfowler commented Mar 24, 2016

@danfowler danfowler closed this Mar 24, 2016

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented Mar 25, 2016

@danfowler where is the support for data import into the DataStore. Does the ckanext-datapackager do this? I thought it only did metadata. Delighted if it does both but good to clarify.

@pwalsh

This comment has been minimized.

Copy link
Member

pwalsh commented Mar 26, 2016

@rgrp you are right, it does not push to datastore. @danfowler can you open an issue on the actual repo for datastore support.

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented Mar 26, 2016

@pwalsh I should emphasize my view that such a thing should almost certainly be a separate mini service (that you integrate) rather than built in to CKAN (data import is non-trivial as we know from OpenSpending: it involves guess, model, validate and then ingest). I think it is definitely best developed as a separate app or service.

The current CKAN integration in terms of a simple API support for getting and setting data package json is great. However, integrating much more would probably not be wise -- first get this working as its own standalone html+json style app (at least on model, validate etc) and then move to integrate (if you want to integrate at all)

@danfowler

This comment has been minimized.

Copy link
Contributor

danfowler commented Mar 29, 2016

@rgrp @pwalsh I made the issue on the other repo, but we can definitely reopen this one if it makes more sense as a separate service.

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented May 2, 2016

Re-opening: clear that we still have some work to do here.

@rufuspollock rufuspollock reopened this May 2, 2016

@rufuspollock rufuspollock changed the title Push (Tabular) Data Packages to CKAN (and data to CKAN DataStore) and Pull (Install) from CKAN [super] CKAN + Tabular Data Packages (Push and Pull) May 2, 2016

@amercader amercader changed the title [super] CKAN + Tabular Data Packages (Push and Pull) [super] CKAN + Tabular Data Packages (Import and Export) May 2, 2016

@rufuspollock

This comment has been minimized.

Copy link
Contributor

rufuspollock commented May 4, 2016

Added notes about existing implmentations and new diagram about CKAN import showing steps involved and overall architecture.

@danfowler danfowler closed this Sep 30, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment