Improve geospatial metadata support into ckan, model, harvester and interface

[ccancellieri]

Spatial support improvements

I'm starting this discussion following a my work to provide improvements over the ckanext-spatial plugin to support iso19115-[1,2,3.2018] as well as ISO19139 into ckan.

ckan/ckanext-spatial#212

I'm personally wondering is we can consider the package model created by the base.py from an iso19139 can be considered the 'standard'.

In particular I'm also thinking if having a set of harvesters to import into ckan geospatial data with a standard de facto package model (extra keys stored into the database) can be considered a goal for ckan.

Trying to follow and extend what has already been developed I'm proposing an implementation of an harvester which is able to harvest data via CSW with different profiles (19115-[*] and iso19139).

The problems I'm encountering are several and I'm proposing some non intrusive pull requests which are a good starting point and are also solving several already opened issues.

ckan/ckanext-spatial#258
ckan/ckanext-spatial#256

Proposal

Here is the pull request for ckan/ckanext-spatial#212 (it requires the 258 and 256 to work on 2.8.x)

ckan/ckanext-spatial#260

One of my goals is to harvest a mix of 19115 (in several versions) as well as 19139 from an existing GeoNetwork shipping around 10000 metadata from 1990 to 2021.

This plugin provides an isolated (all into a subfolder) override of the existing plugin.
I know I can develop that as another extension but the goals are:

1. show how implementing an ISpatialHarvester is almost impossible (several issues discovered into base.py)
2. Trying to reduce by thousands of rows extending and overriding only needed methods (tricky due to very long methods).
3. Improve a ckanx-spatial plugin which looks core to ckan for geospatial metadata but still lacks support for several requests

The TODOS into the spatial_harvester.py and into harvester.py are highlighting the issues I've found, hope someone will take some time to have a look.

Topic

The Model provided by the existing spatial plugin (and iso19115 extension) is core to me since I will have to implement interfaces (based over scheming) to match those models to provide a full migration from GeoNetwork to CKAN to manage geospatial metadata from ckan.

I'm also looking at GeoDcat but I would prefer to have a full iso19115 model before we could start thinking to have also that (there's a xslt from ISO to GeoDcat so once we will have iso in place with serialization it will be easy to provide serialization in GeoDcat-2.0 format as well!)

Questions

Are there other official like implementations of ckanext-spatial plugin (related to iso19115)

[ccancellieri]

I'm creating 2 other integrations
Here is the matching schema:
https://github.com/ccancellieri/ckanext-scheming_iso19115/blob/master/ckanext/scheming_iso19115/scheming/iso.yaml
Here is the spatial_plugin integration overriding (and delegating, see TODO(s)):
https://github.com/ccancellieri/ckanext-spatial/blob/a44750f78dd45a4b0e4d5cbed5f1a87d0bf20f99/ckanext/spatial/harvesters/iso19115/spatial_harvester.py#L1030-L1062
Here I'm extracting the extra fields putting them as dataset fields matching an existing iso19115 dataset format (provided by the iso scheming plugin

The result is an enriched 19139/19115 dataset format (called iso19115) validated and editable via scheming plugin interface.

[ccancellieri]

Hi guys, let me update you on this.

After several attempt I finally have produced a working draft available on the above mentioned repository.
It implements the harvester, the base yaml file and provides a full loop from geonetwork to ckan scheming.

After having proposed this internally we decided that it's not sufficiently flexible and easy to configure to go further.

We will require changes to the schema, multiple text, multiple subfields and several other formats that are simply too far to be implemented in a few with the scheming plugin.

This is why I've abandoned the above plugin in flavour of:
https://github.com/ccancellieri/ckanext-jsonschema

The implementation is still a draft but it works and I'm able to have a quite full model imported directly from geonetwork into ckan.

At the moment I'm focusing over ISO19139 format but the base is the same so soon I'll provide a STAC integration based on this plugin

The produced ISO model is not a GeoDcat but it has been created in collaboration with iso19139 experts and cover almost 90% of the iso fields.

It's interesting to see how in the iso profile, the resources, contacts etc are externalized from the metadata and implemented as resource of a certain type.

Feel free to have a look if interested and let me know.

[ccancellieri]

FYI: we recently released a good draft of this plugin shipping a complete ISO19139 work and including support to:
import (from xml)
export (back to xml)
clone
validate
index (views also)

It can be used in conjunction with the last features added: the views
So we've ported back the terriajs plugin to use the same jsonschema approach so now we can leverage on solr to index a view and a lot more.

I think we could plan a presentation on this, at the moment we are going to document the job done to make it usable.

[disl-pdavid]

As someone who is having to start looking into this soon I applaud your work! I'm very interested to see the documentation and usability.

[ccancellieri]

Thank you so much, fyi: the plugin is already in use since a while. at the moment I'm spending some time trying to document a bit what we've done, but I would encourage you to pull the repo and test it's own so you can start giving feedback.
We have planned to provide a demo thanks to the Monthly meeting since a while, but I'm afraid I would never have that time.
Any feedback is well appreciated.