Export resource(s) to sqlite

[zelima]

We need to export data into a sqlite if requested

Acceptance criteria

• There is SQLite file on S3

Tasks

• do analysis
• function for generating processor (to append to pipeline list)
• edit source spec generator

Analysis

Parameters:

• file-name: optional #defaults to <>.db
• resource-names: [resource-one, reource-two] # required
• table names will be same as resources names (with underscores _)
• mode will always be rewrite (always creates new tables)
  • other options would be appen (append new rows) and update (update if row exists). but let's keep it simple

path on S3: data/sqlite/data/{resourcename}.db

Spec

# request body from CLI
{
  ...
  kind: sqlite
}

Analysis

meta:
  owner: <owner username>
  ownerid: <owner unique id>
  dataset: <dataset name>
  version: 1
  findability: <published/unlisted/private>
inputs:
  -  # only one input is supported atm
    kind: datapackage
    url: <datapackage-url>
    parameters:
      resource-mapping:
        <resource-name-or-path>: <resource-url>
outputs:
  -
    kind: sqlite

when sqlite is in outputs, we need to add two processors:

• dump.to_sql into a temporary file
• add_resource to add that resource to the datapackage (with the proper path and datahub type to indicate it’s a derivative of which resource)

# pipeline-spec
meta:
  ...
inputs:
  - 
    kind: datapackage
    ...
outputs: 
  -
    kind: sqlite

# generator.py in assambler
pipeline = [current_pipeline]
for output in outputs:
    if output[kind] == 'sqlite':
          pipeline.append({run: dump.to_sql, parameters: {engine: lsqlite:///}}})
    etc..

yield pipeline_id, {pipeline: pipeline}

Questions:

What should be path for it?