No description, website, or topics provided.
Python Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
flowmanager
tests
.editorconfig
.gitignore
.travis.yml
Dockerfile
LICENSE
MANIFEST.in
Makefile
README.md
pylama.ini
requirements.dev.txt
requirements.txt
scheduler.py
server.py
setup.py
startup.sh
tox.ini

README.md

DataHQ Spec Store

Build Status

An API server for managing a Source Spec Registry

Quick start

Clone the repo and install

make install

Run tests

make test

Run server

python server.py

Env Vars

  • DATABASE_URL: A SQLAlchemy compatible database connection string (where registry is stored)
  • AUTH_SERVER: The domain name for the authentication server
  • DPP_URL: URL for the datapackage pipelines service (e.g. http://host:post/)

API

Status

/source/{owner}/{dataset-id}/{revision-number}

Note: Also, you can get info about latest and latest successful revisions by hitting following endpoints

  • latest - /source/{owner}/{dataset-id}/latest
  • successful - /source/{owner}/{dataset-id}/successful

Method

GET

Response

{
  "id": "<revision-id>",
  "spec_contents": <source-specifications>,
  "modified": <last-modified>,
  "state": <QUEUED|INPROGRESS|SUCCEEDED|FAILED>,
  "logs": <full-logs>,
  "error_log": [ <error-log-lines> ],
  "stats": {
    "bytes": <number>,
    "count_of_rows": <number>,
    "dataset_name": <string>,
    "hash": <datapackage-hash>
  },
  "pipelines": {
    "<pipeline-id-1>": {
      "title": "Creating CSV",
      "status": "SUCCEEDED",
      "stats": null,
      "error_log": []
    },
    "<pipeline-id-2>": {
      "title": "Creating JSON",
      "status": "INPROGRESS",
      "stats": {},
      "error_log": []
    },
    "<pipeline-id-3>": {
      "title": "Creating ZIP",
      "status": "FAILED",
      "stats": {},
      "error_log": [
        'error',
        'logs'
      ]
    }
  }
}

state definition:

  • QUEUED: Flow created but not running
  • INPROGRESS: Flow is running
  • SUCCEEDED: Finished successfully
  • FAILED: Failed to run

Upload

/source/upload

Method

POST

Headers

  • Auth-Token - permission token (received from conductor)
  • Content-type - application/json

Body

A valid spec in JSON form. You can find example Flow-Spec in README of planer API

Response

{
  "success": true,
  "dataset_id": "<dataset-identifier>",
  "flow_id": "<dataset-identifier-with-revision-number>",
  "errors": [
      "<error-message>"
  ]
}

Update

/source/update

Method

POST

Body

Payload in JSON form.

{
  "pipeline": "<pipeline-id>",
  "event": "queue/start/progress/finish",
  "success": true/false (when applicable),
  "errors": [list-of-errors, when applicable]
}

Response

{
  "success": success/pending/fail,
  "id": "<identifier>"
  "errors": [
      "<error-message>"
  ]
}