Search for Collections not considered (and static servers)?

[uvoges]

Hi,
Is the use case covered in the spec to search for collections (in case of hundreds or thousands of collections) ?

Thanks & regards,
Uwe

[m-mohr]

Yes, I also think this is missing and should be a "building block". We would also be happy to adopt such a building block in STAC and openEO.

[jerstlouis]

I would like to see searching for collections, including by key meta data such as:

• keywords
• title / description
• geospatial & temporal extents
• scale / resolution
• ISO19115 meta data fields

connected to the retrieval of /collections/ , i.e. as filter.

It would be nice if the way this was handled was a building block which connected to this resource paths (to add collections search capabilities, i.e. filtering of results)

Furthermore an end-point which strictly implements /collections/ would in effect act as a catalog, and could catalog collections residing elsewhere.

[bradh]

I could agree keyword, title/description and extent.

Scale / resolution is a non-trivial and not-Common issue. It has completely different meaning in a map vs vector tiles vs coverage vs feature sense.

Linking to ISO19115 feels too specific for Common (possibly a layering violation), and adding that kind of out-reference will make everything more complex (plus expensive if you have to buy the ISO doc). Generic keywords seems more appropriate, although I could maybe see arbitrary key-value metadata pairs.

[jerstlouis]

The correspondence between scale denominator and pixels per meters is quite easily done based on standard equivalence of 0.28 mm / pixels defined in WMS 1.3 ("For the purposes of this International Standard, the common pixel size is defined to be 0,28 mm × 0,28 mm" -- http://portal.opengeospatial.org/files/?artifact_id=14416)

The rationale behind this is that the list of datasets that a user is interested in discovering for a global low resolution map vs. a high detailed street level map is going to be different. Both coverages and feature collections can be rendered together in the same map, so it makes sense to search for data relevant to the scale of interest, and it doesn't matter whether it's raster or vector based, it can still be determined whether that data is relevant for that scale, based on simple scale / resolution equivalence. For tiles, zoom levels for a specific tiling scheme again easily map directly to both a scale and a resolution so that's a non-issue.

What I'm proposing is for a "Search" modular block which can hook onto the base /collections/ listing parameter, and e.g. the ISO metadata fields can be an additional further 'Search/19115Metadata' extension. I am hoping for a nicely integrated modular capability which serves the purpose of CSW, and CSW has a ISO19115 aplication profile ( https://portal.opengeospatial.org/files/?artifact_id=6495 ).

I see the /collections/ as the obvious place for the cataloging functionality, but I don't think this the current focus of OGC API - Records. You can imagine a service which only has /collections/ as a resource, but points to data elsewhere.

[cmheazel]

API-Common supports the standard syntax for URI Key-Value queries. So collections can be searched for using the bbox and datetime parameters as well as Key-Value. Related to this:

• Issue 83 proposes to add the limit parameter to Common, which we expect to add soon
• There are a number of open issues related to additional metadata elements. These would be targets of a key-value query. We need agreement on what (if any) metadata elements should be added

[cmheazel]

@jerstlouis How is a "Search" modular block different from API-Records (CS-W)?

[jerstlouis]

@cmheazel I'm not familiar with the work that's happening in API - Records (and I would like to be, is the current project MetaCat DWG?).

But I am guessing from the little I heard that it is focused on STAC-like results, e.g. returning a GeoJSON list of vector features describing the data pieces, as opposed to hooking up directly to /collections/ as parameters to filtering a list of collections from thousands of collections.

Yet I feel like the purpose of an OGC API - Catalog would be precisely that, the Search extension to /collections/. So perhaps it is two separate things, or perhaps the two could be reconciled?

[cmheazel]

@jerstlouis It's "OGC API - Records SWG" on https://portal.opengeospatial.org/?m=public&orderby=default&tab=7. Telecons are every other Monday at 10:00 Eastern (just after Common). We met this morning so it will two weeks until the next meeting. STAC is a starting point, but I don't expect that we will stop there.

[bradh]

Once again, Commons shouldn't attempt to encompass all of the OGC API - Records (aka CAT4) scope.

[jerstlouis]

Thanks @cmheazel .

Right now from the Overview ( https://github.com/opengeospatial/ogcapi-records ) it returns a feature collection and items, as its own Catalog collection. This is well suited for searching satellite imagery scenes among a few large imagery collections.

I'm arguing the CSW-like functionality we want is this Search functionality right at the /collections/ level, and I have been bringing this up since the London API hackathon, but this is not the direction API - Records is taking right now. The question is whether API - Records should also address that, or if this is another building block (e.g. currently the /collections/ resource is part of Common, I'd argue the /collections/ and /collections/{collectionID} resources should be part of this "API-Catalog").

@bradh I see 'Common' as more the guiding principles that building blocks should be re-used, rather than itself specifying these building blocks. I'm suggesting to move /collections/ out from Common into "API-Catalog" (or API-Records if it is decided to be the same thing). (with a 'Core' that is just the simple listing, and extensions supporting e.g. search parameters, hierarchy, ISO 19115 profile...)

[cmheazel]

@jerstlouis Keep in mind that I put the GitHub up this afternoon. Much of it is plagiarized, oops, re-used, from other OGC API specifications. There is still a lot of work to do and changes to make.

[cmheazel]

@jerstlouis Take a look at API-Features. Each extension is a separate specification with it's own conformance class(s). These are our building blocks. Extensions to Common can take the same approach.

BTW: the CQL extension is the Common Query Language, first defined in OGC Catalog 1.0. Something we may want to leverage in API-Records.

[cmheazel]

@jerstlouis One more thing on API-Records. Check the pull request, not the README.

[jerstlouis]

@cmheazel I filed this issue opengeospatial/ogcapi-records#20 in API-Record

[cmheazel]

API-Common-Collections supports filtering on the /collections request using the bbox and datetime parameters. Derivative standards can add search terms as appropriate.

Recommend that this issue be closed.

[jerstlouis]

@cmheazel Query parameters on collections should be in a separate conformance class, to facilitate implementation by static servers. As done in Coverages: https://github.com/opengeospatial/ogc_api_coverages/tree/master/standard/requirements/geodata-filteredList .

[cmheazel]

How do we handle the case where a server does not (or cannot) support all of the required parameters on an operation:

1. Allow the server to advertise which parameters are supported. This should be a simple advertisement placed within the context of the operation.
2. Add a new conformance class for each parameter (or logical bundles of parameters)
3. Define default behaviors for each parameter, what should the server do if it does not support that parameter
4. Don't allow

[jyutzler]

My ordered preference (high to low): 1/3/2/4

[m-mohr]

What parameters we speak about specificly? bbox and datetime?
It it's just them, I think I'd be for 4. Or we'd need to make /queryables (from CQL) core. I don't think we should introduce yet another way to advertise "queryables".

[jerstlouis]

@m-mohr query parameters on /collections (parameters on the list of collections itself, such as bbox and datetime)

1. Meant some additional mechanism which does not require clients to fully parse an OpenAPI document, so as not to inherit that full complexity
2. (my preference) Means e.g. a geodata-filteredList conformance class which lets the client know that the server supports query parameters on /collections (e.g. a static server simply hosted on Apache probably would not)
3. Likely means to allow the server to ignore query parameters it does not understand or support
4. Means forcing servers to support query parameters (excluding most static servers, and a bad idea in my opinion)

[m-mohr]

So what's the difference between /collections and /collections/.../items? Features doesn't have a separate conformance class for the query parameters, too. Introducing them for collections only seems inconsistent.

[jerstlouis]

@m-mohr part of the question is how can a client reliably know what query parameters are supported, without parsing an OpenAPI? 2. is suggesting conformance classes as part of that solution.

[m-mohr]

Yes, understood. My take (might be provocative to some) is that it should support at least some basics (at least geospatial queries, better also include temporal) otherwise it doesn't feel it should be an "Open Geospatial Consortium" API implementation.

If we make everything conformance classes, what do we need /api (OpenAPI parsing) for? It just doesn't feel very useful is mostly everything is optional. And it also feels that it gets too bloated / hard to implement...

Just my 2ct

[jerstlouis]

@m-mohr

Queries are great if the server supports it, but if the data to host is reasonably small, why prevent putting up files on static storage (which might be much more affordable and easier to put together than a full-blown service supporting query parameters) from being able to comply to the OGC API?

What feels too bloated / hard to implement to me is a client being forced to parse OpenAPI documents to do anything with a service.

That brings in way more complexity for a client being written in a framework where OpenAPI tools are not the norm or non-existent, to simply access an otherwise simple OGC API.

[m-mohr]

Queries are great if the server supports it, but if the data to host is reasonably small, why prevent putting up files on static storage (which might be much more affordable and easier to put together than a full-blown service supporting query parameters) from being able to comply to the OGC API?

If support for static catalogs should be possible, then it should also be implemented in Features for Items.

What feels too bloated / hard to implement to me is a client being forced to parse OpenAPI documents to do anything with a service.

Fully agreed. I actually prefer something like conformance classes as I never liked the approach to parse complex OpenAPI definitions and I'd likely never implement it. I'd completely drop the OpenAPI parsing approach from OGC APIs...

[jerstlouis]

@m-mohr I would equally hope that putting a GeoJSON file at /collections/{collectionId}/items that does not accept query parameters could conform to Features as well (unfortunately in Features right now I believe this is all within the 'core' conformance class, yet I do recall static servers coming up many times in discussions...).

[ogc-leedahl]

Not to mention that it is very easy for an Open API document to become out of sync with links on a landing page. In Testbed 16 we had a case where the Open API document said the link to a service was X and the landing page said the link was Y. Which one am I supposed to follow? QGIS for example, followed the landing page which was incorrect in this case. I had to talk to the participant responsible for the OGC API – Features implementation to have both fixed so they pointed to the same place. I know that this is technically an implementation bug; however, it does illustrate the potential for conflicts between the Open API, landing pages and conformance classes. Michael From: Jerome St-Louis <notifications@github.com> Reply-To: opengeospatial/oapi_common <reply@reply.github.com> Date: Monday, September 7, 2020 at 7:33 AM To: opengeospatial/oapi_common <oapi_common@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [opengeospatial/oapi_common] Search for Collections not considered ? (#69) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. @m-mohr<https://urldefense.us/v2/url?u=https-3A__github.com_m-2Dmohr&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=NQECgj9eMVDZOOjk8htmUHG97lPsawpW5GIU2Y_70pE&m=behsmOB5lBKjgS8uVIH8xTRKxL0iW_BzSY2BIXaixK0&s=JOnOi3IpvAXE_SJo8Ko7PJiKw3IXQqj38yYRvY74FEQ&e=> Queries are great if the server supports it, but if the data to host is reasonably small, why prevent putting up files on static storage (which might be much more affordable and easier to put together than a full-blown service supporting query parameters) from being able to comply to the OGC API? What feels too bloated / hard to implement to me is a client being forced to parse OpenAPI documents to do anything with a service. That brings in way more complexity for a client being written in a framework where OpenAPI tools are not the norm or non-existent, to simply access an otherwise simple OGC API. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.us/v2/url?u=https-3A__github.com_opengeospatial_oapi-5Fcommon_issues_69-23issuecomment-2D688328863&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=NQECgj9eMVDZOOjk8htmUHG97lPsawpW5GIU2Y_70pE&m=behsmOB5lBKjgS8uVIH8xTRKxL0iW_BzSY2BIXaixK0&s=ENIUOVvdZ3SjCGqhOLRjjpV0tMCQPu_3JCsD6TWIT0A&e=>, or unsubscribe<https://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AJEJIJMITGF6K6O5PB7WE63SETOIHANCNFSM4JIPSD3Q&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=NQECgj9eMVDZOOjk8htmUHG97lPsawpW5GIU2Y_70pE&m=behsmOB5lBKjgS8uVIH8xTRKxL0iW_BzSY2BIXaixK0&s=bgy6rwVB_hAD5AUbyGB-WDh19l2SI6yCM_Ud01wiQpA&e=>.

[ghobona]

2020-09-15 OGC API SWGs Cross Cutting Issues session during OGC Member Meeting

The following resolution was agreed upon during the session:

• Agreement that we do not want to impose requirements on servers that do not support Static Web Server capabilities.
• Not imposing requirements on all OGC API standards.
• We can do this on Common Core.

[ghobona]

2020-09-15 OGC API Cross Cutting Issues session during OGC Member Meeting

See the related resolution at #85 (comment)

[m-mohr]

What does this mean in detail to the questions raised here?

[ghobona]

@m-mohr

Some transcribed notes from the discussion are below.

If we want to have parameters optional that means we may have to have conformance classes down to the individual parameter.

Technically that would be correct. For example, you could define the Bbox parameter as a conformance class, then other APIs would be able to use the Bbox.

It may actually be harder to test it because it is not a resource but just another resource.

Before we go down that route, we should discuss this and get feedback from developers – especially client developers.

We should keep in mind that these are just edge cases. The specs already cover 80% of the likely use cases.

[cmheazel]

/req/core/query-param-unknown
/per/core/query-param-specified
/per/core/query-param-tolerance
/req/core/query-param-invalid
and
/per/core/query-param-value-tolerance

Provide servers considerable leeway in how they advertise acceptable parameters and values. They also provide servers leeway in determining if a client supplied parameter/value is acceptable. I think this is as far as we can go.

Recommend closing.

[jerstlouis]

I still think we have work to do before we can close this issue:

1. Have a separate conformance classes for filtering /collections by the query parameters defined in Common - Part 2 (e.g. BBOX, DateTime) so that static servers can be implemented that do not need to support any query parameters on /collections which is of little value if the API only offers a handful of collections. I believe Features - Part 1 did not define those parameters at /collections either, so another advantage of having this as a separate conformance class is not to break Features.
2. Clarify the relationship with OGC API - Records - Part 2. I think it defines those query parameters for filtering /collections as initially wished for in this issue.
3. Clarify the relationship with the emerging metadata defined e.g. in 2DTMS & TileSet Metadata (Table 14). The ablity to filter on these properties would be very useful. There is the tangential question on where that extra information could be available (e.g. directly as additional properties of the collection resource?).
4. Potentially have a conformance class giving the ability to filter collections by properties from standard (e.g. ISO 19115) geospatial metadata available linked by data-meta.

[cmheazel]

June 26, 2021 - make the processing of bbox, datetime, and limit a separate conformance class.
Re: 3 and 4 - potential use cases for the registry approach to modular API standards.

[cmheazel]

The Simple Query Conformance Class in Part 2 specifies the use of the bbox, datetime, and limit parameters against a /Collections resource. The Core Conformance Class in Part 1 specifies how filtering on arbitrary name-value pairs should be done and how that capability should be advertised. The Part 1 Core Conformance Class is recommended for implementations of Common Part 2.