configurable outputs formats - zip, sqlite, more...

[zelima]

We need to export data into a zip, sqlite, other rel db, etc... if requested

• Define updated source spec: see analysis section
  • Decide on defaults (what do we generate automatically and what is the algorithm?)
• zip dumper (already exists - ....)
• Implement support for this new source spec in backend (assembler) - Export datapackage to zip datopian/assembler#19
  • Plan out what will happen e.g. where will files live on disk, how will you parse things etc
    • path: data/{export_type}/data/{filenam}.ext Eg:
    • data/zip/data/myreource.zi, or data/sqlite/data/myreource.db
  • Document in README
• Update CLI to support this (TODO: define how this should work) Set output formats: zip, sqlite datopian/data-cli#165
• Add in frontend - Zip/sqlite etc output formats datopian/frontend#105
• Write a brief blog post / doc about this

Extras:

• sqlite dumper (??) - TODO - new issue in PM or relevant repo just for this ... Export resource(s) to sqlite datopian/assembler#15
• sql dumper (??) - Will work in same way as sqlite dumper but with extra info (engine URI, credentials).

Analysis

Current request from CLI is following

{
  meta: { owner: ..., ownerid: ..., dataset: ..., version: ... }
  inputs: [ {kind: datapackage, url: ..., parameters: { ... }} ]
}

The new request may include outputs parameters. By default, outputs always contain "csv" and "json" so the previous request did not need to include it (only for tabular).

Outputs is the list of formats ("kinds") to export data to. Eg new request may look like this:

{
  meta: { owner: ..., ownerid: ..., dataset: ..., version: ... }
  inputs: [ {kind: datapackage, url: ..., parameters: { ... }} ]
  outputs: [ {kind: zip, paramaters: {out-file: my-awsome.zip }}, {kind: sqlite}, etc..] 
}

available kinds, parameters and defaults

kinds: sqlite, zip, rdbms

dump.to_sql needs the following parameters:

parameters: {
'tables': {
  'tablename': {
    'resource-name': 'sample-resource',
    }
 }
}
}

parameters for sqlite: None for now

parameters for zip:

• out-file: optional #defaults to <<dataset-name>>.zip

parameters for rdbms: ????

• file-name: optional #defaults to <<dataset-name>>.db
• resource-names: [resource-one, reource-two] # required
• engine: 'DB URI'
• credentials: {accecckey: ..., secret_key}
  • why would somebody ever want to provide credentials for us?

## Specifications

```yaml
meta:
  owner: <owner username>
  ownerid: <owner unique id>
  dataset: <dataset name>
  version: 1
  findability: <published/unlisted/private>
inputs:
  -  # only one input is supported atm
    kind: datapackage
    url: <datapackage-url>
    parameters:
      resource-mapping:
        <resource-name-or-path>: <resource-url>
outputs: // backend has some rule about what is auto-added for you e.g. csv and json is always there for tabular input
  -
    kind: csv 
  -
    kind: json
    resources: '2017-data'
  -
    kind: rdbms  (dump.to_sql)
    parameters:
      engine: <pointer to ...>
  -
    kind: sqlite (dump.to_sql)
  -
    kind: npm  (New processor (not yet implemented))
    parameters:
      credentials: <pointer to ...>
-
    kind: zip (dump.to_zip)
    parameters: 
        out-file: name of the file

[zelima]

FIXED. While there's only zip supported as output, closing as we decided to polish the existing product and hold on new features