[FEATURE][MetaSearch] add support for OGC API - Records

[tomkralidis]

Description

This PR updates MetaSearch to provide support for the OGC API - Records (OARec) standard. As part of the OGC API evolution, OGC API - Records is the successor to OGC:CSW (like OGC API - Features is to WFS, etc.).

This is a new feature worked on by @pvgenuchten and myself that does not need backporting. Note both Paul and I are part of the OGC API - Records Standards Working Group.

The changes in this PR basically abstract MetaSearch to interact with an abstracted search backend concept (CSW, or OGC API - Records, and can be extended as needed). As a result we can now more easily add search APIs much easier (OpenSearch, STAC API, etc.). CSW functionality is unaffected as works as expected.

The PR is ready for review, but why the WIP status? The OGC API - Records standard is currently in development and the current timeline for a public RFC by OGC is Q1 2022. The question here is whether folks are happy with putting this in master. What do folks think? I've added those who have been involved/interested in MetaSearch in the past given this significant enhancement to MetaSearch.

From my perspective, I would support adding to master and updating the documentation, specifying the draft status of the specification so users are aware that the specification may change a bit more over the next few months. We can even update the label of the "OGC API - Records" Catalog Type select box (see screenshot 2 below) to "OGC API - Records (draft)" if that helps. We will need to flush out how actionable links are handled over the next few months (which should be a significant improvement compared to the CSW publish/find/bind workflow).

Adding this feature to QGIS is a huge win for easier search and will help both users and providers for the long term. As always I'm happy to maintain and keep this up to date.

Screenshots below:

OARec service description:

OARec service/catalog type:

OARec Raw API request/response rendering update:

[nyalldawson]

I can't see any issues with the code changes, but I don't have any understanding of the new API and can't comment on that part of it :)

[nyalldawson]

(nice work by the way!!)

[alexbruy]

I'm afraid I'm not the right person to review this as I did not follow development of the owslib and don't have any understanding of the new API. Code looks ok for me, thought I sill see a lot of dependencies which duplicate native functionality, e.g. xml syntax highlighting.

[tomkralidis]

Note: this PR needs OWSLib 0.25.0 for the OARec updates (MetaSearch will still work in master for CSW mode, but not for OARec mode). @jef-n / others: any idea on when the lastest release of OWSLib will be available? Is this tied to UbuntuGIS updates?

[agiudiceandrea]

any idea on when the lastest release of OWSLib will be available?

It seems python3-owslib-0.25.0-1 is already available in the OSGeo4w v2 repository.

[jef-n]

@jef-n / others: any idea on when the lastest release of OWSLib will be available? Is this tied to UbuntuGIS updates?

-rw-r--r-- 1 jef users 124383 Aug 23 21:18 /osgeo/download/osgeo4w/v2/x86_64/release/python3/python3-owslib/python3-owslib-0.25.0-1.tar.bz2

But that's just osgeo4w. I don't do the Debian/Ubuntu dependencies.

owslib is also available in debian unstable & testing: https://tracker.debian.org/news/1253381/owslib-0250-1-migrated-to-testing/

[tomkralidis]

Thanks @jef-n. Once this PR gets an approving review I will merge and update docs thereafter.

[pvgenuchten]

having ogc-api-records in qgis supports also the development of server implementations (like geonetwork, pycsw, qgis server) they can use qgis metasearch to test their implementation. +1 to merge

[timlinux]

Great update @tomkralidis !

@nyalldawson It was a long time ago that we discussed @tomkralidis taking over MetaSearch maintainership, haven't been able to dig up the thread, but it is great to see MetaSearch getting some much needed love! Is there anything blocking us from going ahead and merging this? I guess he is his own best reviewer of his work :-P

[nyalldawson]

My only reservation is the bump in library dependency. I think before we merge someone should do a quick scan of the different platforms (at least mac, debian, ubuntu) and check which versions are available there. Otherwise we'd need to ensure there's a graceful fallback approach for older versions. (Keeping in mind the fallout from #37175, which was the last time we bumped the owslib dependency!)

[tomkralidis]

Thanks @nyalldawson. cc @kalxas for input on packaging.

[kalxas]

We can definitely backport OWSLib 0.25 to UbuntuGIS repositories if that helps.

[github-actions]

@tomkralidis
This pull request has been tagged as requiring documentation.

A documentation ticket will be opened at https://github.com/qgis/QGIS-Documentation when this PR is merged.

Please update the description (not the comments) with helpful description and screenshot to help the work from documentors.
Also, any commit having [needs-doc] or [Needs Documentation] in will see its message pushed to the issue, so please be as verbose as you can.

Thank you!

[tomkralidis]

Thanks @nyalldawson. Rebased.

[tomkralidis]

@nyalldawson I think the check failures are unrelated?

[github-actions]

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

• that all unit tests are passing

• that all comments by reviewers have been addressed

• that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes

  • add a description of workflows which this pull request fixes

  • add screenshots if applicable

• that you have written unit tests where possible
  In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
  If there is no further activity on this pull request, it will be closed in a week.

[github-actions]

While we hate to see this happen, this PR has been automatically closed because it has not had any activity in the last 21 days. If this pull request should be reconsidered, please follow the guidelines in the previous comment and reopen this pull request. Or, if you have any further questions, just ask! We love to help, and if there's anything the QGIS project can do to help push this PR forward please let us know how we can assist.

[github-actions]

@tomkralidis
A documentation ticket has been opened at qgis/QGIS-Documentation#7141
It is your responsibility to visit this ticket and add as much detail as possible for the documentation team to correctly document this change.
Thank you!

[nyalldawson]

@tomkralidis

The official master docker images are raising an error now:

File \"/usr/share/qgis/python/plugins/MetaSearch/dialogs/maindialog.py\", line 49, in <module><br>&nbsp; &nbsp; from MetaSearch.dialogs.newconnectiondialog import NewConnectionDialog<br>&nbsp; File \"/usr/lib/python3/dist-packages/qgis/utils.py\", line 888, in _import<br>&nbsp; &nbsp; mod = _builtin_import(name, globals, locals, fromlist, level)<br>&nbsp; File \"/usr/share/qgis/python/plugins/MetaSearch/dialogs/newconnectiondialog.py\", line 34, in <module><br>&nbsp; &nbsp; from MetaSearch.search_backend import CATALOG_TYPES<br>&nbsp; File \"/usr/lib/python3/dist-packages/qgis/utils.py\", line 888, in _import<br>&nbsp; &nbsp; mod = _builtin_import(name, globals, locals, fromlist, level)<br>ModuleNotFoundError: No module named 'MetaSearch.search_backend'

Do you know what is causing this?

[tomkralidis]

@nyalldawson this should be fixed in #46285

[agiudiceandrea]

@tomkralidis it seems xmldialog.ui file was not renamed apidialog.ui, while it is referenced in apidialog.py as apidialog.ui (which is nonexistent)

BASE_CLASS = get_ui_class('apidialog.ui')

[tomkralidis]

thanks @agiudiceandrea; PR in #46292

[agiudiceandrea]

Hi @tomkralidis, please see "MetaSearch OGC API Records is not accepting authenticated endpoint credentials" #55979.