Storing additional metadata for a dataset beyond the default metadata in CKAN is a common scenario. CKAN provides a simple way to do this by allowing you to store arbitrary key/value pairs against a dataset when creating or updating the dataset. These appear under the "Additional Information" section on the web interface and in 'extras' field of the JSON when accessed via the API.
Default extras can only take strings for their keys and values, no validation is applied to the inputs and you cannot make them mandatory or restrict the possible values to a defined list. By using CKAN's IDatasetForm plugin interface, a CKAN plugin can add custom, first-class metadata fields to CKAN datasets, and can do custom validation of these fields.
In this tutorial we are assuming that you have read the
/extensions/tutorial
When a dataset is created, updated or viewed, the parameters passed to CKAN (e.g. via the web form when creating or updating a dataset, or posted to an API end point) are validated against a schema. For each parameter, the schema will contain a corresponding list of functions that will be run against the value of the parameter. Generally these functions are used to validate the value (and raise an error if the value fails validation) or convert the value to a different value.
For example, the schemas can allow optional values by using the ~ckan.lib.navl.validators.ignore_missing validator or check that a dataset exists using ~ckan.logic.validators.package_id_exists. A list of available validators and converters can be found at the validators and converters. You can also define your own validators for custom validation.
We will be customizing these schemas to add our additional fields. The :py~ckan.plugins.interfaces.IDatasetForm interface allows us to override the schemas for creation, updating and displaying of datasets.
~ckan.plugins.interfaces.IDatasetForm.create_package_schema ~ckan.plugins.interfaces.IDatasetForm.update_package_schema ~ckan.plugins.interfaces.IDatasetForm.show_package_schema ~ckan.plugins.interfaces.IDatasetForm.is_fallback ~ckan.plugins.interfaces.IDatasetForm.package_types
CKAN allows you to have multiple IDatasetForm plugins, each handling different dataset types. So you could customize the CKAN web front end, for different types of datasets. In this tutorial we will be defining our plugin as the fallback plugin. This plugin is used if no other IDatasetForm plugin is found that handles that dataset's type.
The IDatasetForm also has other additional functions that allow you to provide a custom template to be rendered for the CKAN frontend, but we will not be using them for this tutorial.
Create a new plugin named ckanext-extrafields and create a class named ExampleIDatasetFormPlugins inside ckanext-extrafields/ckanext/extrafields/plugins.py that implements the IDatasetForm interface and inherits from SingletonPlugin and DefaultDatasetForm.
../../ckanext/example_idatasetform/plugin_v1.py
The :py~ckan.plugins.interfaces.IDatasetForm.create_package_schema function is used whenever a new dataset is created, we'll want update the default schema and insert our custom field here. We will fetch the default schema defined in :py~ckan.logic.schema.default_create_package_schema by running :py~ckan.plugins.interfaces.IDatasetForm.create_package_schema's super function and update it.
../../ckanext/example_idatasetform/plugin_v1.py
The CKAN schema is a dictionary where the key is the name of the field and the value is a list of validators and converters. Here we have a validator to tell CKAN to not raise a validation error if the value is missing and a converter to convert the value to and save as an extra. We will want to change the :py~ckan.plugins.interfaces.IDatasetForm.update_package_schema function with the same update code.
../../ckanext/example_idatasetform/plugin_v1.py
The :py~ckan.plugins.interfaces.IDatasetForm.show_package_schema is used when the :py~ckan.logic.action.get.package_show action is called, we want the default_show_package_schema to be updated to include our custom field. This time, instead of converting to an extras field, we want our field to be converted from an extras field. So we want to use the :py~ckan.logic.converters.convert_from_extras converter.
../../ckanext/example_idatasetform/plugin_v1.py
The :py~ckan.plugins.interfaces.IDatasetForm.package_types function defines a list of package types that this plugin handles. Each package has a field containing its type. Plugins can register to handle specific types of packages and ignore others. Since our plugin is not for any specific type of package and we want our plugin to be the default handler, we update the plugin code to contain the following:
../../ckanext/example_idatasetform/plugin_v1.py
In order for our new field to be visible on the CKAN front-end, we need to update the templates. Add an additional line to make the plugin implement the IConfigurer interface
../../ckanext/example_idatasetform/plugin_v2.py
This interface allows to implement a function :py~ckan.plugins.interfaces.IDatasetForm.update_config that allows us to update the CKAN config, in our case we want to add an additional location for CKAN to look for templates. Add the following code to your plugin.
../../ckanext/example_idatasetform/plugin_v2.py
You will also need to add a directory under your extension directory to store the templates. Create a directory called ckanext-extrafields/ckanext/extrafields/templates/ and the subdirectories ckanext-extrafields/ckanext/extrafields/templates/package/snippets/.
We need to override a few templates in order to get our custom field rendered. Firstly we need to remove the default custom field handling. Create a template file in our templates directory called package/snippets/package_metadata_fields.html containing
../../ckanext/example_idatasetform/templates/package/snippets/package_metadata_fields.html
This overrides the custom_fields block with an empty block so the default CKAN custom fields form does not render. Next add a template in our template directory called package/snippets/package_basic_fields.html containing
../../ckanext/example_idatasetform/templates/package/snippets/package_basic_fields.html
This adds our custom_text field to the editing form. Finally we want to display our custom_text field on the dataset page. Add another file called package/snippets/additional_info.html containing
../../ckanext/example_idatasetform/templates/package/snippets/additional_info.html
This template overrides the default extras rendering on the dataset page and replaces it to just display our custom field.
You're done! Make sure you have your plugin installed and setup as in the extension/tutorial. Then run a development server and you should now have an additional field called "Custom Text" when displaying and adding/editing a dataset.
Before we continue further, we can clean up the :py~ckan.plugins.interfaces.IDatasetForm.create_package_schema and :py~ckan.plugins.interfaces.IDatasetForm.update_package_schema. There is a bit of duplication that we could remove. Replace the two functions with:
../../ckanext/example_idatasetform/plugin_v3.py
If you need to add a custom field where the input options are restricted to a provided list of options, you can use tag vocabularies /tag-vocabularies. We will need to create our vocabulary first. By calling ~ckan.logic.action.vocabulary_create. Add a function to your plugin.py above your plugin class.
../../ckanext/example_idatasetform/plugin_v4.py
This code block is taken from the example_idatsetform plugin. create_country_codes tries to fetch the vocabulary country_codes using ~ckan.logic.action.get.vocabulary_show. If it is not found it will create it and iterate over the list of countries 'uk', 'ie', 'de', 'fr', 'es'. For each of these a vocabulary tag is created using ~ckan.logic.action.create.tag_create, belonging to the vocabulary country_code.
Although we have only defined five tags here, additional tags can be created at any point by a sysadmin user by calling ~ckan.logic.action.create.tag_create using the API or action functions. Add a second function below create_country_codes
../../ckanext/example_idatasetform/plugin_v4.py
country_codes will call create_country_codes so that the country_codes vocabulary is created if it does not exist. Then it calls ~ckan.logic.action.get.tag_list to return all of our vocabulary tags together. Now we have a way of retrieving our tag vocabularies and creating them if they do not exist. We just need our plugin to call this code.
Update :py~ckan.plugins.interfaces.IDatasetForm._modify_package_schema and :py~ckan.plugins.interfaces.IDatasetForm.show_package_schema
../../ckanext/example_idatasetform/plugin_v4.py
We are adding our tag to our plugin's schema. A converter is required to convert the field in to our tag in a similar way to how we converted our field to extras earlier. In :py~ckan.plugins.interfaces.IDatasetForm.show_package_schema we convert from the tag back again but we have an additional line with another converter containing :py~ckan.logic.converters.free_tags_only. We include this line so that vocab tags are not shown mixed with normal free tags.
Add an additional plugin.implements line to to your plugin to implement the :py~ckan.plugins.interfaces.ITemplateHelpers, we will need to add a :py~ckan.plugins.interfaces.ITemplateHelpers.get_helpers function defined for this interface.
../../ckanext/example_idatasetform/plugin_v4.py
Our intention here is to tie our country_code fetching/creation to when they are used in the templates. Add the code below to package/snippets/package_metadata_fields.html
../../ckanext/example_idatasetform/templates/package/snippets/package_metadata_fields.html
This adds our country code to our template, here we are using the additional helper country_codes that we defined in our get_helpers function in our plugin.
In order to customize the fields in a resource the schema for resources needs to be modified in a similar way to the packages. The resource schema is nested in the package dict as package['resources']. We modify this dict in a similar way to the package schema. Change _modify_package_schema to the following.
../../ckanext/example_idatasetform/plugin.py
Update :py~ckan.plugins.interfaces.IDatasetForm.show_package_schema similarly
../../ckanext/example_idatasetform/plugin.py
Save and reload your development server CKAN will take any additional keys from the resource schema and save them the its extras field. The templates will automatically check this field and display them in the resource_read page.
Now that we've added our custom field, we can customize the CKAN web front end search page to sort datasets by our custom field. Add a new file called ckan/example_idatasetform/templates/package/search.html containing:
../../ckanext/example_idatasetform/templates/package/search.html
This overrides the search ordering drop down code block, the code is the same as the default package search block but we are adding two additional lines that define the display name of that search ordering (e.g. Custom Field Ascending) and the SOLR sort ordering (e.g. custom_text asc). If you reload your development server you should be able to see these two additional sorting options on the dataset search page.
The SOLR sort ordering can define arbitrary functions for custom sorting, but this is beyond the scope of this tutorial for further details see http://wiki.apache.org/solr/CommonQueryParameters#sort and http://wiki.apache.org/solr/FunctionQuery
You can find the complete source for this tutorial in the ckanext/example_idatasetform directory.