Merge pull request #4030 from ckan/3989-data-dictionary-docs

Data Dictionary Docs
ckan · Feb 22, 2018 · 211f5b2 · 211f5b2
2 parents 10634dd + 43139d6
commit 211f5b2
Show file tree

Hide file tree

Showing 2 changed files with 99 additions and 42 deletions.
diff --git a/ckanext/datastore/templates/datastore/snippets/dictionary_form.html b/ckanext/datastore/templates/datastore/snippets/dictionary_form.html
@@ -1,16 +1,25 @@
 {% import 'macros/form.html' as form %}
 
 <h3>{{ _( "Field {num}.").format(num=position) }} {{ field.id }} ({{ field.type }})</h3>
-          {{ form.select('info__' ~ position ~ '__type_override',
-            label=_('Type Override'), options=[
-            {'name': '', 'value': ''},
-            {'name': 'text', 'value': 'text'},
-            {'name': 'numeric', 'value': 'numeric'},
-            {'name': 'timestamp', 'value': 'timestamp'},
-            ], selected=field.get('info', {}).get('type_override', '')) }}
-          {{ form.input('info__' ~ position ~ '__label',
-            label=_('Label'), id='field-f' ~ position ~ 'label',
-            value=field.get('info', {}).get('label', ''), classes=['control-full']) }}
-          {{ form.markdown('info__' ~ position ~ '__notes',
-            label=_('Description'), id='field-d' ~ position ~ 'notes',
-            value=field.get('info', {}).get('notes', '')) }}
+
+{#
+  Data Dictionary fields may be added this snippet. New fields following
+  the 'info__' ~ position ~ '__namegoeshere' convention will be saved
+  as part of the "info" object on the column.
+#}
+
+{{ form.select('info__' ~ position ~ '__type_override',
+  label=_('Type Override'), options=[
+  {'name': '', 'value': ''},
+  {'name': 'text', 'value': 'text'},
+  {'name': 'numeric', 'value': 'numeric'},
+  {'name': 'timestamp', 'value': 'timestamp'},
+  ], selected=field.get('info', {}).get('type_override', '')) }}
+
+{{ form.input('info__' ~ position ~ '__label',
+  label=_('Label'), id='field-f' ~ position ~ 'label',
+  value=field.get('info', {}).get('label', ''), classes=['control-full']) }}
+
+{{ form.markdown('info__' ~ position ~ '__notes',
+  label=_('Description'), id='field-d' ~ position ~ 'notes',
+  value=field.get('info', {}).get('notes', '')) }}
diff --git a/doc/maintaining/datastore.rst b/doc/maintaining/datastore.rst
@@ -233,6 +233,49 @@ alongside CKAN.
 To install this please look at the docs here: http://docs.ckan.org/projects/datapusher
 
 
+.. _data_dictionary:
+
+---------------
+Data Dictionary
+---------------
+
+DataStore columns may be described with a Data Dictionary. A Data Dictionary tab
+will appear when editing any resource with a DataStore table.
+The Data Dictionary form allows entering the following values for
+each column:
+
+* **Type Override:** the type to be used the next time DataPusher is run to load
+  data into this column
+* **Label:** a human-friendly label for this column
+* **Description:** a full description for this column in markdown format
+
+Extension developers may add new fields to this form by overriding the default
+Data Dictionary form template ``datastore/snippets/dictionary_form.html``.
+
+The Data Dictionary is set through the API as part of the :ref:`fields` passed
+to :meth:`~ckanext.datastore.logic.action.datastore_create` and
+returned from :meth:`~ckanext.datastore.logic.action.datastore_search`.
+
+
+.. _dump:
+
+---------------------
+Downloading Resources
+---------------------
+
+A DataStore resource can be downloaded in the `CSV`_ file format from ``{CKAN-URL}/datastore/dump/{RESOURCE-ID}``.
+
+For an Excel-compatible CSV file use ``{CKAN-URL}/datastore/dump/{RESOURCE-ID}?bom=true``.
+
+Other formats supported include tab-separated values (``?format=tsv``),
+JSON (``?format=json``) and XML (``?format=xml``). E.g. to download an Excel-compatible
+tab-separated file use
+``{CKAN-URL}/datastore/dump/{RESOURCE-ID}?format=tsv&bom=true``.
+
+.. _CSV: https://en.wikipedia.org/wiki/Comma-separated_values
+
+
+
 -----------------
 The DataStore API
 -----------------
@@ -279,23 +322,6 @@ API reference
    :members:
 
 
-.. _dump:
-
-Download resource
------------------
-
-A DataStore resource can be downloaded in the `CSV`_ file format from ``{CKAN-URL}/datastore/dump/{RESOURCE-ID}``.
-
-For an Excel-compatible CSV file use ``{CKAN-URL}/datastore/dump/{RESOURCE-ID}?bom=true``.
-
-Other formats supported include tab-separated values (``?format=tsv``),
-JSON (``?format=json``) and XML (``?format=xml``). E.g. to download an Excel-compatible
-tab-separated file use
-``{CKAN-URL}/datastore/dump/{RESOURCE-ID}?format=tsv&bom=true``.
-
-.. _CSV: https://en.wikipedia.org/wiki/Comma-separated_values
-
-
 .. _fields:
 
 Fields
@@ -304,22 +330,40 @@ Fields
 Fields define the column names and the type of the data in a column. A field is defined as follows::
 
     {
-        "id":    # a string which defines the column name
+        "id":  # the column name (required)
         "type":  # the data type for the column
+        "info": {
+            "label":  # human-readable label for column
+            "notes":  # markdown description of column
+            "type_override":  # type for datapusher to use when importing data
+            ...:  # other user-defined fields
+	}
     }
 
-Field **types are optional** and will be guessed by the DataStore from the provided data. However, setting the types ensures that future inserts will not fail because of wrong types. See :ref:`valid-types` for details on which types are valid.
+Field types not provided will be guessed based on the first row of provided data.
+Set the types to ensure that future inserts will not fail because of an incorrectly
+guessed type. See :ref:`valid-types` for details on which types are valid.
+
+Extra ``"info"`` field values will be stored along with the column. ``"label"``,
+``"notes"`` and ``"type_override"`` can be managed from the default :ref:`data_dictionary`
+form.  Additional fields can be stored by customizing the Data Dictionary form or by
+passing their values to the API directly.
 
 Example::
 
     [
         {
-            "id": "foo",
-            "type": "int4"
+            "id": "code_number",
+            "type": "numeric"
         },
         {
-            "id": "bar"
-            # type is optional
+            "id": "description"
+            "type": "text",
+            "info": {
+                "label": "Description",
+                "notes": "A brief usage description for this code",
+                "example": "Used for temporary service interruptions"
+            }
         }
     ]
 
@@ -331,19 +375,21 @@ Records
 A record is the data to be inserted in a DataStore resource and is defined as follows::
 
     {
-        "<id>":  # data to be set
-        # .. more data
+        column_1_id: value_1,
+        columd_2_id: value_2,
+        ...
     }
 
 Example::
 
     [
         {
-            "foo": 100,
-            "bar": "Here's some text"
+            "code_number": 10,
+            "description": "Submitted successfully"
         },
         {
-            "foo": 42
+            "code_number": 42,
+            "description": "In progress"
         }
     ]
 
@@ -437,8 +483,10 @@ name
 oid
     The PostgreSQL object ID of the table that belongs to name.
 
+
+-------------------
 Extending DataStore
-===================
+-------------------
 
 Starting from CKAN version 2.7, backend used in DataStore can be replaced with custom one. For this purpose, custom extension must implement `ckanext.datastore.interfaces.IDatastoreBackend`, which provides one method - `register_backends`. It should return dictonary with names of custom backends as keys and classes, that represent those backends as values. Each class supposed to be inherited from `ckanext.datastore.backend.DatastoreBackend`.