Add schemas for ckanext-scheming for DCAT-AP 2.1

[amercader]

We could provide one or two predefined schemas covering all DCAT and DCAT-AP fields ready to be used or customized by instance maintainers.

This will also help in handling multilingual metadata (#55)

We could have:

• Basic schema with most common fields (eg mandatory and recommended properties in DCAT-AP)
• Full schema with all fields (that can later be slimmed down)
• Presets file with validators if needed

What we need:

• JSON schema files in the relevant location
• API and functional tests
• Docs

[ted-strauss-K1]

I am using ckanext-scheming to define schemas for special dataset types.
Link to example schema: https://github.com/ckan/ckanext-scheming/blob/master/ckanext/scheming/camel_photos.json

I would like to have the metadata of these data types automatically represented in DCAT via this extension. I'm pretty sure that is the scope described by this issue, but just clarifying.

Currently, when I append .jsonld or .rdf to the dataset it displays only the normal metadata and ignores the other fields. Do those field need to have a namespace defined for them somehow?

Is there a recommended way to achieve this functionality?

[wood-chris]

I think this is what I've been looking for (and hoping someone else would have done). When I first saw that this extension was compliant with dcat-ap, I assumed that it meant that it would automagically create the relevant fields (with the required label for properties that are mandatory in the AP) - but it seems like I was a bit optimistic!

Does one already exist? If not, it looks like I'll need to create one - if it's helpful to add to a repo of common schemas (that may or may not already exist) I'd be happy to add it when I've done it

[metaodi]

@wood-chris there is a scheming mapping for DCAT-AP Switzerland (which is a slightly adapted version of DCAT-AP EU) here: https://github.com/opendata-swiss/ckanext-switzerland/blob/master/ckanext/switzerland/dcat-ap-switzerland_scheming.json

[amercader]

Dumping some thoughts here on scheming support.

At the time the processors (parsers/serializers) that map between CKAN and DCAT were written, usage of ckanext-scheming was still starting to become widespread, so custom DCAT fields that didn't link directly to standard CKAN fields were stored as extras (see all the ones marked extra: here). So the DCAT version_notes field would be stored as:

{
    "name": "test_dataset_dcat",
    "title": "Test dataset DCAT",
    // ....
    "extras": [
         {"key": "version_notes", "value": "Some version notes"}
         // ....
    ]
}

The pattern nowadays is to create custom fields in a scheming schema, that internally handles the conversion to / from extras:

{
    "name": "test_dataset_dcat",
    "title": "Test dataset DCAT",
    "version_notes": "Some version notes"
    // ....
}

That's the goal of creating a DCAT scheming schema, that all properties are custom fields of the CKAN Dataset, aligning with the dcat:Dataset ones.

The difficulty here is how to offer support for existing sites using previous versions of the extension.

• For the CKAN -> DCAT (Serialization) direction it should be fine. The serializers support both forms, and for a given field (like version_notes) will check first for a root level field in the dataset_dict and if it's not there, look for an extra with that key (here)
• For the DCAT -> CKAN (Parsing) direction, we will find issues. As I said, the current parsers will store the custom fields in extras, which will make the resulting dataset dict incompatible with a scheming schema that defines the field as a dataset field. When creating or updating, we will get the following error:

  File "/home/adria/dev/pyenvs/ckan-py3.9/ckan/ckan/logic/action/create.py", line 185, in package_create
    raise ValidationError(errors)
ckan.logic.ValidationError: None - {'extras': [{'key': ['There is a schema field with the same name']}]}

The second case is relevant to CKAN sites that import DCAT RDF representations from other systems (I'd imagine that in most cases through the DCAT harvester) and create CKAN datasets from them.

My current thinking on how to approach this is:

• Change the default parsers so they can store DCAT fields as root fields rather than extras, keeping the old behaviour via config option [1]
• This will go into a new major ckanext-dcat version (2.0.0), with clear documentation:
  • New sites using scheming will just need to use the included schema and everything should work
  • New sites that don't want to use scheming (unlikely) will have to set the config option, otherwise the custom fields will be ignored
  • Existing sites that have custom DCAT profiles and want to use scheming will have to store custom fields as root level fields (or drop any custom validators/logic they are currently using now to make dcat and scheming work together)
  • Existing sites that they don't want to use scheming and keep the old behaviour, they just need to set the config option [1]

[1] At first I thought about being clever and inspecting the dataset schema (if scheming was being used) to see if there were DCAT fields defined, and not store the values in extras if so but that seemed brittle, and sites could have different schemas in used, potentially for different DCAT versions even. I think it's better to be explicit and make site maintainers

Sorry if this is a bit convoluted, here's a TLDR:

In the next major version of ckanext-dcat I want to change the RDF DCAT Parsers so they store custom DCAT fields as first level CKAN dataset fields instead of dataset extras, but keep the old behviour via config option for backwards compatibility

[amercader]

Chatting with @wardi about this, actually using scheming_dataset_schema_show to check if there is a schema that contains DCAT fields could be a really good approach, as besides detecting if values need to be stored at the root level, we could use it to mark DCAT fields with certain keys in the schema (like dcat_validators, etc). These are available to the output of scheming_dataset_schema_show, snippets, validators etc.
To the point of knowing what schema to use a good approach might be:

1. One explicitly provided when creating the Profile class: for instance for sites with harvesters that have more than one dataset schema that could be a harvester config option
2. If scheming is loaded, default to the dataset schema
3. If scheming is not loaded or there is no schema defined, fall back to store things in extras

[amercader]

PR with summary of work done so far here: #281

[amercader]

Done in #281 + #288