Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

DataHQ Spec Store

Build Status

An API server for managing a Source Spec Registry

Quick start

Clone the repo and install

make install

Run tests

make test

Run server

python server.py

Env Vars

  • DATABASE_URL: A SQLAlchemy compatible database connection string (where registry is stored)
  • AUTH_SERVER: The domain name for the authentication server
  • DPP_URL: URL for the datapackage pipelines service (e.g. http://host:post/)

API

Status

/source/{owner}/{dataset-id}/{revision-number}

Note: Also, you can get info about latest and latest successful revisions by hitting following endpoints

  • latest - /source/{owner}/{dataset-id}/latest
  • successful - /source/{owner}/{dataset-id}/successful

Method

GET

Response

{
  "id": "<revision-id>",
  "spec_contents": <source-specifications>,
  "modified": <last-modified>,
  "state": <QUEUED|INPROGRESS|SUCCEEDED|FAILED>,
  "logs": <full-logs>,
  "error_log": [ <error-log-lines> ],
  "stats": {
    "bytes": <number>,
    "count_of_rows": <number>,
    "dataset_name": <string>,
    "hash": <datapackage-hash>
  },
  "pipelines": {
    "<pipeline-id-1>": {
      "title": "Creating CSV",
      "status": "SUCCEEDED",
      "stats": null,
      "error_log": []
    },
    "<pipeline-id-2>": {
      "title": "Creating JSON",
      "status": "INPROGRESS",
      "stats": {},
      "error_log": []
    },
    "<pipeline-id-3>": {
      "title": "Creating ZIP",
      "status": "FAILED",
      "stats": {},
      "error_log": [
        'error',
        'logs'
      ]
    }
  }
}

state definition:

  • QUEUED: Flow created but not running
  • INPROGRESS: Flow is running
  • SUCCEEDED: Finished successfully
  • FAILED: Failed to run

Upload

/source/upload

Method

POST

Headers

  • Auth-Token - permission token (received from conductor)
  • Content-type - application/json

Body

A valid spec in JSON form. You can find example Flow-Spec in README of planer API

Response

{
  "success": true,
  "dataset_id": "<dataset-identifier>",
  "flow_id": "<dataset-identifier-with-revision-number>",
  "errors": [
      "<error-message>"
  ]
}

Update

/source/update

Method

POST

Body

Payload in JSON form.

{
  "pipeline": "<pipeline-id>",
  "event": "queue/start/progress/finish",
  "success": true/false (when applicable),
  "errors": [list-of-errors, when applicable]
}

Response

{
  "success": success/pending/fail,
  "id": "<identifier>"
  "errors": [
      "<error-message>"
  ]
}

About

No description, website, or topics provided.

Resources

License

Releases

No releases published

Languages

You can’t perform that action at this time.