[epic] v0.2 Development of the Client SDK

[rufuspollock]

v0.1 was about basic "push" functionality which is done.

v0.2 is about refactoring this to be more elegant and support for reading/writing metadata

API design should follow https://github.com/datopian/ckan-client-guide (but tweaked for js/nodejs idioms)

https://github.com/okfn/ckan.js could probably be reused for the plumbing layer.

Acceptance

• client.push(datasetMetadata) function works which takes dataset JS object (in f11s format) and puts it to CKAN datastore
• client.pushBlob(resource, onProgressFn): uploads a resource's data to the Blob Storage. resource is a data.js File object of a specific type e.g. FileInterface
• client.retrieve(datasetIdOrName)

Porcelain

• client.action(...)

Bonus

• client.pushResource(resource, dataset-name) takes resource in data.js and saves data to storage and metadata
• client.pushResourceMedata

Tasks

• client.action: wrapper for CKAN action API Method to call CKAN action API #10
• client.push: put dataset metadata Push dataset metadata client.push(datasetMetadata)  #11
• client.pushBlob: refactor existing code pushBlob method to push resource data to blob storage #12
• client.retrieve: retrieve method for dataset metdata #7

API design

This is indicative - see individual issues for precise specifications

Uses data.js and its Frictionless (f11s) data interface as the way to access files and datasets. Metadata follows Frictionless formats.

Plumbing

This is relatively raw low level APIs.

• Storage: getting authz, interaction with giftless etc
• MetaStore: CKAN action API v3

client = Client(...)
client.action_api(...)

Porcelain

// create a dataset (or overwrite existing one) - just puts metadata (file uploading is separate)
client.push(datasetMetadata)

// longform of previous item
resource = open(path) # f11s.Resource object
// this uploads just the data to storage ...
// resource has .stream, 
client.pushBlob(resource)
// f11s.Dataset({metadata})
// so can do dataset.descriptor
dataset = client.retrieve(dataset_name)
dataset.resources.push(resource.descriptor)
client.push(dataset)

// bonus
// convenience function
// dataset must exist
client.pushResource(f11s_resource, dataset=dataset_name)
// convenience
client.pushResourceMetadata(...) # only save resource metadata ... 

// GET functions
client.retrieve(dataset_id_or_name)

// BONUS
client.retrieveResource(resource_id OR dataset_id + resource_name)

Example usage

import data.js as f11s
import ckanclient

// resource is a data.js file object from a browser file selection
resource = filereader object wrapped ...


dataset = {
  descriptor:
    name: ... 
    resources:
}

What is an f11s_dataset?

dataset.descriptor => Frictionless formatted object for a dataset ...

put

Client.action_api = async function(action, payload) {

}

Client.put = async function(f11s_dataset) {
  ckan_metadata = f11s2ckan.dataset(f11s_dataset.descriptor)
  // https://docs.ckan.org/en/2.9/api/index.html#ckan.logic.action.create.package_create
  // see existing code https://github.com/okfn/ckan.js#catalog
  await res = client.action_api('package_create', ckan_metadata)
  return res
}

[mariorodeghiero]

• refactoring this to be more elegant
  • I created this issue to do it: Refactor sdk to be more elegant  #5

[mariorodeghiero]

• Created tests and the action function to be able to create_resource and update_resource.

[mariorodeghiero]

In Progress:

• Refactoring Datapub to use the create_resource and update_resource
  Estimate: 0.5day