Skip to content

Commit

Permalink
[#255] Remove "Using the Data API" from Datastore docs, add section a…
Browse files Browse the repository at this point in the history 
…bout internal database structure
  • Loading branch information
@domoritz
domoritz committed Feb 11, 2013
1 parent d5eb41b commit 85b5b11
Showing 1 changed file with 22 additions and 169 deletions.
191 changes: 22 additions & 169 deletions doc/datastore-api.rst
View file
Expand Up @@ -39,6 +39,8 @@ requests, parameters should be provided as JSON objects.

.. note:: Lists can always be expressed in different ways. It is possible to use lists, comma separated strings or single items. These are valid lists: ``['foo', 'bar']``, ``'foo, bar'``, ``"foo", "bar"`` and ``'foo'``. Additionally, there are several ways to define a boolean value. ``True``, ``on`` and ``1`` are all vaid boolean values.

.. note:: The table structure of the DataStore is explained in :ref:`db_internals`.

.. automodule:: ckanext.datastore.logic.action
:members:

Expand Down Expand Up @@ -128,12 +130,12 @@ You can find more information about the formatting of dates in the `date/time ty

.. _date/time types section of the PostgreSQL documentation: http://www.postgresql.org/docs/9.1/static/datatype-datetime.html

.. _resource_aliases:

Table aliases
-------------

A resource in the DataStore can have multiple aliases that are easier to remember than the resource id. Aliases can be created and edited with the datastore_create API endpoint. All aliases can be found in a special view called ``_table_metadata``.
Resource aliases
----------------

A resource in the DataStore can have multiple aliases that are easier to remember than the resource id. Aliases can be created and edited with the :meth:`~ckanext.datastore.logic.action.datastore_create` API endpoint. All aliases can be found in a special view called ``_table_metadata``. See :ref:`db_internals` for full reference.

.. _datastore_search_htsql:

Expand All @@ -158,178 +160,29 @@ The DataStore supports querying with multiple API endpoints. They are similar bu
**Ease of use** Easy Complex Medium
**Flexibility** Low High Medium
**Query language** Custom (JSON) SQL HTSQL
**Connect multiple resources** No Yes Not yet
**Join resources** No Yes Not yet
**Use aliases** Yes Yes Yes
============================== ======================================================== ============================================================ =============================


Using the Data API
==================

Before going into detail about the API and the examples, it is useful to create a resource first so that you can store data against it in the Datastore. This can be done either through the CKAN Graphical User Interface or the API.

Using the data API, we need to send JSON with this structure::

{
id: Uuid,
name: Name-String,
title: String,
version: String,
url: String,
resources: [ { url: String, format: String, description: String, hash: String }, ...],
author: String,
author_email: String,
maintainer: String,
maintainer_email: String,
license_id: String,
tags: Tag-List,
notes: String,
extras: { Name-String: String, ... }
}

To the following endpoint:

* Dataset Create Endpoint: ``http://{YOUR-CKAN-INSTALLATION}/api/action/package_create``

More details about creating a resource through the Data API are available on the :ref:`CKAN API page <api>`.


Examples
--------

.. note:: There is a special view that lists all available resources from the DataStore. It can be found at the alias ``_table_metadata``.

Some of the following commands require obtaining an :ref:`API Key <get-api-key>`.

cURL (or Browser)
~~~~~~~~~~~~~~~~~

The following examples utilize the cURL_ command line utility. If you prefer, you you can just open the relevant urls in your browser::

# This creates a datastore
curl -X POST <ENDPOINT:datastore_create> -H "Authorization: <YOUR-API-KEY>" -d '{"resource_id": "<RESOURCE-ID>", "fields": [ {"id": "a"}, {"id": "b"} ], "records": [ { "a": 1, "b": "xyz"}, {"a": 2, "b": "zzz"} ]}'


#This queries a datastore
curl "<ENDPOINT:datastore_search>?resource_id=<RESOURCE-ID>" -H "Authorization: <YOUR-API-KEY>"

.. _cURL: http://curl.haxx.se/

Javascript
~~~~~~~~~~

A simple ajax (JSONP) request to the data API using jQuery::

$.ajax({
url: '<ENDPOINT:datastore_search>',
data: {'resource_id': '<RESOURCE-ID>'},
dataType: 'jsonp',
success: function(data) {
alert('Total results found: ' + data.result.total)
}
});

The Data API supports CORs so you can also write to it (this requires the json2_ library for ``JSON.stringify``)::

Coming soon...

..
The Data API supports CORs so you can also write to it (this requires the json2_ library for ``JSON.stringify``)::
var data = {
title: 'jones',
amount: 5.7
};
$.ajax({
url: {{endpoint}},
type: 'POST',
data: JSON.stringify(data),
success: function(data) {
alert('Uploaded ok')
}
});

.. _json2: https://github.com/douglascrockford/JSON-js/blob/master/json2.js

Python
~~~~~~

A Python URLLib2 datastore_create and datastore_search would look like::

#! /usr/bin/env python
import urllib
import urllib2
import json

auth_key = '<YOUR-API-KEY>'

# In python using urllib2 for datastore_create it is...

url = "<API-ENDPOINT>"

datastore_structure = {
'resource_id': '<RESOURCE-ID>',
'fields': [{"id": "a"}, {"id": "b"}],
"records": [{"a": 12, "b": "abc"}, {"a": 2, "b": "zzz"}]
}
headers = {'content-type': 'application/json', 'Authorization': auth_key}

req = urllib2.Request(url + 'datastore_create', data=json.dumps(datastore_structure), headers=headers)
response = urllib2.urlopen(req)


# in python for datastore_search using urllib2....

datastore_structure = {
'resource_id': '<RESOURCE-ID>'
}

url_values = urllib.urlencode(datastore_structure)
req = urllib2.Request(url + 'datastore_search?' + url_values, headers=headers)
response = urllib2.urlopen(req)

print response.read()

print "done\n"


Using the Python Requests_ library we can create a datastore like this::

#! /usr/bin/env python

import requests
import json

auth_key = '<YOUR-API-KEY>'

url = '<API-ENDPOINT>'

datastore_structure = {
'resource_id': '<RESOURCE-ID>',
'fields': [ {"id": "a"}, {"id": "b"} ],
"records": [ { "a": 1, "b": "xyz"}, {"a": 2, "b": "zzz"} ]
}
headers = {'content-type': 'application/json', 'Authorization': auth_key}
r = requests.post(url + 'datastore_create', data=json.dumps(datastore_structure), headers=headers)
print "done, and now for a quick search\n"

datastore_structure = {
'resource_id': '<RESOURCE-ID>'
}
headers = {'content-type': 'application/json', 'Authorization': auth_key}
r = requests.post(url + 'datastore_search', data=json.dumps(datastore_structure), headers=headers)

print r.text

print "done\n"
.. _db_internals:

Internal structure of the database
----------------------------------

.. _Requests: http://docs.python-requests.org/
The DataStore is a thin layer on top of a PostgreSQL database. Each DataStore resource belongs to a CKAN resource. The name of a table in the DataStore is always the resource id of the CKAN resource for the data.

PHP
~~~~~~
As explained in :ref:`resource_aliases`, a resource can have mnemonic aliases which are stored as views in the database.

::
All aliases (views) and resources (tables respectively relations) of the DataStore can be found in a special view called ``_table_metadata``. To access the list, open ``http://{YOUR-CKAN-INSTALLATION}/api/action/datastore_search?resource_id=_table_metadata``.

Coming soon...
``_table_metadata`` has the following fields:

_id
Unique key of the relation in ``_table_metadata``.
alias_of
Name of a relation that this alias point to. This field is ``null`` iff the name is not an alias.
name
Contains the name of the alias if alias_of is not null. Otherwise, this is the resource id of the CKAN resource for the DataStore resource.
oid
The PostgreSQL object ID of the table that belongs to name.

0 comments on commit 85b5b11

Please sign in to comment.