Collection of Collections

[cmheazel]

Can a collection contain other collections? The GFM says yes, but that may be too much complexity to address in simple core.

[cportele]

See also opengeospatial/ogcapi-features#90 and opengeospatial/ogcapi-features#76

[cportele]

If we are talking about collections of collections we should also make sure we define the distinction between dataset (as in DCAT, see https://www.w3.org/TR/vocab-dcat/#vocabulary-overview) and our feature/coverage/etc collection resources. Datasets have a specific role (see schema.org, the Google Dataset Search, the Data on the Web Best Practices) and we should fit into this picture.

In WFS 3.0 Core we support the simplest case, "every WFS instance provides access to a single dataset" (see e.g. the Scope statement).

In the strawman OGC API resource diagram in which I have added the feature/process/coverage/tile/style resources that I found in drafts I have also included one approach how to support multiple dataset as an input for discussion.

[nmtoken]

For coverages we have collection of coverages, is that a similar concept?

[jerstlouis]

I believe it would be extremely useful to define a proper optional collection of collections conformance class as part of OGC API - Common (Collections).

See also discussions on recursive datasets and large number of collections in #17, #69 and #111 .

[cmheazel]

A comment received from Uwe Voges:
I was wondering how the use case will be covered that there may be hierarchies of collections (see also STAC) ?
This will include the case that some hierarchy levels may be searchable for items (records) while others may not ...
Example:

Meteosat Catalogue (Searchable for all Meteosat products)
MTG (Searchable just for MTG products)
MSG (Searchable just for MSG)
Metop Catalogue (Searchable for all Metop products)
EPS (Searchable for EPS products)
EPS-SG (Searchable for EPS-SG products)
Sentinel Catalogue (not searchable)
Sentinel3 Catalogue (Searchable for Sentinel3 products)

Ok, one idea could be to provide different API´s with an own landing page etc for every hierarchyLevel....

Another option could be to have the /collections resource which may not even return links to single collections (/collection) but also links to other hierarchy levels of collections (/collections) (rel="collections" like the rel "data" in the landing page).

I tried ad hoc a JSON encoding of the example above which could be provided by the link from the landing page (rel="data") (just a few metadata elements considered here):

{
"collections": [
{
"id": "msgCat",
"type": "catalogue",
"title": "Meteosat Catalogue",
"links": [
{
"href": "http://.../meteosatCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. MTG, MSG. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
},
{
"href": "http://.../meteosatCat/collections/meteosat",
"rel": "collection",
"title": "At this endpoint you can retrieve a description of this catalogue as well as hypermedia controls that allow you to query the catalogue."
}
]
},
{
"id": "metopCat",
"type": "catalogue",
"title": "Metop Catalogue",
"links": [
{
"href": "http://.../metopCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. EPS-SG. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
},
{
"href": "http://.../metopCat/collections/metop",
"rel": "collection",
"title": "At this endpoint you can retrieve a description of this catalogue as well as hypermedia controls that allow you to query the metop catalogue."
}
]
},
{
"id": "sentinel3Cat",
"type": "catalogue",
"title": "Sentinel3 Catalogue",
"links": [
{
"href": "http://.../sentinelCat/collections",
"rel": "collections",
"title": "At this endpoint you can retrieve hypermedia controls that allow you to navigate sub-collections, e.g. Sentinel3. Here you can retrieve descriptions of these sub-catalogues as well as hypermedia controls that allow you to query the sub-catalogues"
}
]
}
]
}

[cmheazel]

Copied from the Wiki at the request of Dave Blodgett

One of the most contentious issues we face in API development is consensus on standard resource paths. API-Features uses /collections/{collectionId}/items. Other SWGs plan to use /collections/{collectionId}/coverage, /collections/{collectionId}/maps, etc. Which is correct?

There appears to be consensus on /collections/{collectionId}. At least for those resources which are typically distributed as collections. The final part of the path tells you something about the collection. A taxonomy of these path fragments could look something like this:

/coverage -> Measurements organized in a regular or irregular grid

/map -> 2D Rendered information displayable for human visual consumption

/tile -> A subset of a dataset based on area of interest and level of detail

/items -> Any

Wait a minute -- "Any"!!

What use is a path fragment which can be anything?

Actually, it’s the most useful of all.

OGC Web Services are built around a taxonomy of data types; maps, features, coverages, etc. It’s a data provider centric view of the world. But what if we took a user-centric view of APIs. What if collections are built around the information that a user needs to do their job?

Consider a first responder. They are going to need a building model, routes of ingress and egress, photographs, overhead imagery, utilities, an elevation model, and all sorts of other supporting data. Everything they need to respond to an incident at a specific location. And they are going to want it in one neat little package, a collection. So how is our first responder going to use “/collections/110_Maple_Avenue/”? /maps, /coverage, /routes all have value. But how do you access the collection contents as a whole? /items.

/items allows us to locate and access any content in the collection regardless of type. So the fundamental access path for any collection is /collections/{collectionId}/items. This does not mean that the other paths don’t have value, just at a different level.

We can view /coverage, /maps, etc. as profiles of /items. They both restrict and extend /items. They restrict in that they only provide access to a specific type of item in the collection. They extend in that they bring to the user a whole collection of tools which are only useful for that type of resource. So “/collections/110_Maple_Avenue/coverage” brings to the first responder all of the sub-setting and processing tools which come with coverage data. But only for that subset of the collection which is coverage data.

So what does /items actually return? A Feature collection.

Now that the outrage has had a chance to settle,

We tend to think of Features as vector data. But at its most fundamental level, a Feature is an empty container. It has neither type, identity, nor properties. So a Feature can grow up to be just about anything. Which is exactly the sort of construct we are looking for.

Under this approach, API-Common would define the bbox, limit, and datetime parameters and how they would be applied to a generic data item. Other API standards would specialize those capabilities for the unique properties of the resources they address.

[rob-metalinkage]

when a problem is very complex in one level of abstraction - as I feel this one is - it can help to change the domain of discourse to a different level (e.g. fourier transforms between time and frequency domains)

In this case I think the problem is too narrow a focus, and hence different solutions for smuggling additional information into the places you can see..

At a higher level of abstraction I believe you can look at "context" for metadata - and understand there is always an element of "out-of-band" metadata - the question is what is necessary and what is acceptable.

In the "classic" SOA a client may go to a catalog and discover a service end-point and out-of-band metadata about the service type - at which point they use this context to formulate a query (get capabilities) about additional metadata the server will self-report.

The same pattern holds true for data descriptions. To what extent is the data responsible for self-reporting. In general, as little as possible, otherwise you need to find and access and interrogate too much before you can select useful data.

So much of the context comes down to some form of "out-of-band" description - some form of catalog. This can vary from a hyper-local (hardcoded in a client) catalog, to service self-description to aggregated catalogs with well-known service interfaces. Its still pushing the data description problem to some out-of-band metadata construct.

So now the question is really one of minimising impedance when a client accesses such out-of-band sources of context - i.e. the interoperability of the description. This includes a description of how the data model of the data maps onto the data meta-model of the service (in this case collections/items/features)

As @cmheazel suggests - you have all the mechanisms you need - its a choice of how you use them, and the point here IMHO is understanding how to map interoperable metadata forms onto the two different models supported by the service: container based addressing and reporting of metadata (capabilities). We can push all this to the issue of publishing Feature Types with sufficient richness and interoperability that such mappings can be immediately inferred - but its probably worth developing test cases and examples for what ever pattern you choose.

[dblodgett-usgs]

I thought I responded to this but my comment is missing.

@cmheazel : When I called out the wiki post, and said "collection" of collection issues, I thought it was a pun!

I linked to the collection TAG trying to make the suggestion that I thought your wiki post was a good frame for bringing some closure to a lot of similar issues related to using collections. Not just this one. But I'll take it. We need to get this one settled!

[jerstlouis]

The approach currently implemented in our client & service:

• /collections returns a list of a all collections

• /collections/{collectionId} returns the resource for that collection (containing the links to the rel items, tiles etc.)

• As an extension, a special character within the {collectionId} is understood to be a hierarchy delimiter, and the service supports returning a partial list of collection if the client requests /collections/{partial-collectionId}.

This works nicely with multi-layer vector tiles as well, in which case a {partial-collectionId} can be both a collection linking to multi-layer /tiles end-point, as well as list the individual collections.

At the moment that special character happens to be a '/', and the invalidity of this can be debated, but in practice it has caused no interoperability problem with our service during 2+ years of TIEs.
This could be changed to use another character and specify that character with a property, e.g. "hierarchySeparator" : "-".

Features also state that an OGC API end-point should contain a single dataset, and that hierarchies of multiple datasets should be implemented entirely at another level (i.e. higher than the landing page).

I propose that combined hierarchies of datasets and collections could be implemented as an integrated optional capability, while still being able to mark which level constitutes a dataset. There could also potentially be landing pages at different levels...

[dblodgett-usgs]

@jerstlouis -- I'm having a hard time connecting the first part of your comment to the last. Maybe some more for instance API paths would help?

Are you saying we might have a hierarchy of OGC-API landing pages?

What API path would dereference to a dataset (landing page) and what API path would dereference to a collection that is subordinate to that dataset?

I think you are saying that an integrated dataset-landing page might be something like:
/collections/demo/ would behave as an integrated collection with endpoints like:
/collections/demo/tiles and /collections/demo/items?

Subordinate to that integrated collection could then be the members at API paths like:
/collections/demo-roads/ which would dereference to a dataset (landing page) for demo-roads?

Is this what you are getting at or am I misunderstanding?

[jerstlouis]

@dblodgett-usgs The first part of my comment we have implemented and used in many TIEs (but currently with slashes rather than dashes).
e.g.

• http://maps.ecere.com/geoapi/collections
• http://maps.ecere.com/geoapi/collections/vtp
• http://maps.ecere.com/geoapi/collections/vtp/Daraa2
• http://maps.ecere.com/geoapi/collections/vtp/Daraa2/AgricultureSrf

When requesting the JSON representation of these, the whole list of collections is returned, but the HTML representation only shows the immediate hierarchy level.

The latter part of my comment about the landing page at each level, I guess would imply combining those list of collections with the elements normally found in a landing page (e.g. having an api description, in addition to a list of collections and/or tiles & items)

[dblodgett-usgs]

OK, I think I follow where you are going but some example API paths that illustrate the concept with some functional description would be helpful. Thanks for the live example links, but again, some functional description of what these are would be a bit help.

[jerstlouis]

http://maps.ecere.com/geoapi/collections -- This is the list of all datasets/collections on the server
http://maps.ecere.com/geoapi/collections/vtp -- This is a filtered list of datasets/collections under "vtp"
http://maps.ecere.com/geoapi/collections/vtp/Daraa2 -- This is a list of collections for the dataset "vtp/Daraa2" (something would need to identify it as this point in the hierarchy being a single dataset)
http://maps.ecere.com/geoapi/collections/vtp/Daraa2/AgricultureSrf -- This is the AgricultureSrf collection inside the "vtp/Daraa2" dataset.

I was suggesting we could possibly define such an extension of combined hierarchy of datasets & collections, so that both datasets and collections can have an arbitrary number of hierarchy levels, sharing the same hierarchical mechanism.

Does that help? Not sure what you're asking as "functional description".

[dblodgett-usgs]

Yes it does -- to a point. The flexible typing of end points is really confusing and API path templates would help.

I feel like the root /collections isn't needed at all. If you want an ad-hoc cataloging mechanism, then that's what you need.

Why not drop the collections path literal all together and have a GNOSIS dataset catalog that links to landing pages for each of the datasets you want to mash up? The hierarchy could be documented however meets your use case.

You would end up with a flat collection of API end points (datasets) that could be orchestrated into new API endpoints.

e.g.
http://maps.ecere.com/geoapi/vtp (which is a mash up of others)
http://maps.ecere.com/geoapi/Daraa2 (which is a mash up of others)
http://maps.ecere.com/geoapi/AgricultureSrf (which is just a dataset)

All of which would have various functions available -- {root}/collections for feature collections, {root}/tiles for tiles, {root}/coverages for coverages.

The API path complexity just doesn't help, IMHO.

[jerstlouis]

@dblodgett-usgs I don't see how this provides the desired functionality of being able to browse both datasets and collections in a hierarchical manner?

There is a need for both hiearchies of collections (if a dataset is inherently organized in a hierarchical manner), and of datasets (to categorize very large number of datasets).

This is what I am proposing here (with the possibility of combining both), but what you just suggested is to have a flat list of datasets, and a flat list of collections.

[dblodgett-usgs]

... it doesn't handle the complexity -- that's a feature, not a bug.

We are working on common building blocks not a comprehensive SDI. We need building blocks that allow us to create complexity, not complexity that happens to be composed of building blocks.

If we get a good set of building blocks and start seeing common patterns of complexity, maybe we encode them as best practices or even standard implementation patterns, but IMHO, we need to pursue simple building blocks first and foremost that get us to a baseline rather than wandering around aimlessly pretending we can handle the complexity of these kinds of mash up implementations before we have even shown we can handle the simple use cases.

[jerstlouis]

@dblodgett-usgs

The desire for hierarchical collections, as an extension, is what this issue is about.

I am not sure I understand your point. A lot of simple and more complex building blocks have been shown to work, but we are still side-stepping this important issue.

[dblodgett-usgs]

Hold on... "what is a collection" is not settled yet so hierarchical collections as an extension is debating about uncertainty -- ill-defined concepts and functionality.

What I'm getting at is that there is a path forward here where we can focus on building simple components [that can be assembled into hierarchies] that can be composed into systems.

If, once we've learned how to crawl (i.e. get the components right), and experimented sufficiently with how to run (i.e. build systems and hierarchies), we have some implementation patterns that are worth standardizing, we can and should.

Pretending that we can make proposals for shared integrated practices, without well defined components that are being integrated isn't productive. What I'm trying to do is find productive paths forward to build well-defined, well-separated, simple, building blocks that allow us to get to complexity.

[jerstlouis]

@dblodgett-usgs Well, "what is a collection" is debated in many other separate issues, including #111, #122, this new interesting one #140...

This one was about hierarchy. While I agree that the basics should be settled first, sometimes I like to keep believing that this was settled in Toulouse, as both the published Features standard, and the draft Coverage specifications support a collection base path and providing "rel" : "items" links to vector and/or raster data at:

/collections/{collectionID}

[jeffharrison]

The prototype OGC API for the 3D Pilot is advancing for many Participants, TIEs are underway and demos today for the first time showed heirarchies.

The heirarchies approach being tested in the 3D Pilot largely emulates existing OGC practices with respect to volume hierarchies, and seeks to make it accessible for a wide variety of content types, scales and granularities (as OGC Staff indicated). A participating component can implement TIE functionality or not, but it is part of the hands-on collaborative prototyping process.

In a larger context and to paraphrase OGC Staff, it's fine that OGC API Features decided to go with a flat list of collections to emphasize simplicity where appropriate. But not all OGC API's are, nor are intended to be, extended versions of OGC API Features. Not all building blocks of OGC API Common need to be, or even should be, extensions of OGC API Features.

There is no Common Collections heirarchies guidance that went missing ...it's under development now and I welcome the opportunity to work on it.

Best Regards,
Jeff

[jeffharrison]

I understand you may feel the approach recently discussed with respect to 'children' in hierarchies may not be compatible with other approaches, but I think it's OK for a Pilot to test different approaches, assess results and document findings for the community.

If you feel you didn't have input into something, let's bring it up to the project team and get it sorted out. I'm not exactly sure what that is in reference to.

Best Regards,
Jeff

[jerstlouis]

@pvretano Well, it's just a question of how those conformance classes should be organized. I would find it useful to have a Part for conformance classes that facilitate handling a large number of collections, but that's just my own opinion :)

About parts organization, I also still wonder how CITE will handle that, will each part get its own test suite and compliance status? That is probably my biggest concern with super simple Parts (in addition to the boilerplate / overhead, and difficulty for implementers to discover and refer to all the different specification documents).

[pvretano]

@jerstlouis agreed - the "large number of collections handling" stuff should be in its own Part ... not Part 2 of common which should be limited to the basic /collections behaviour. I would go so far as to say that bbox, datetime and limit should not be in Part 2 but that ship may have sailed.

As for CITE, it has nothing to do with parts; it has to do with requirements and conformance classes. The Features CITE test for example (at least the beta) checks the conformance declaration and depending on what is in there, it triggers the appropriate requirements tests. So, if your server declares http://www.opengis.net/spec/ogcapi-features-2/1.0/conf/crs (i.e. Part 2 of Features) in its conformance declaration then CITE will run all the tests that check all the requirements for that conformance class.

I would be surprised if there was a specific CITE test for Common because common is something that is referenced by other specifications like Features, Records, etc. and testing common's requirements will be part of testing conformance to the containing specification. So if your server conforms to Feature core then it also conforms to common since common is a prerequisite for Features core (well not right at the moment but some later version of Features will reference Common). Put another way, your cannot get your server "certified" as compliant to common. You get your server as "certified" compliant to Feature Core or Records Core and common is one set of requirements test that your server has to pass to get that certification.

[jerstlouis]

@pvretano

I would go so far as to say that bbox, datetime and limit should not be in Part 2 but that ship may have sailed.

That is the Part 2 "Simple Query" conformance class that I was referring to, that I also think could go in there.

Thanks for the CITE clarifications, that makes sense. So if I understand correctly, there will be only one CITE test (testing all parts with tests implemented so far) / single certification fee to conform to Features and all its parts.

So any new "Common" part that come up, could automatically be picked up by the other specs to test and report conformance to those additional requiments (e.g. simple query at /collections, Records queries at /collections, hierarchical collections...), if they apply based on existing dependencies (e.g. dependency on Common collections).

Cool!

[pvretano]

@jerstlouis yikes! I am not sure about fees or anything like that. You will need to ping OGC for that information.

[ghobona]

We use a single Executable Test Suite (ETS) to check for compliance to conformance classes from both Part 1 and Part 2.

Compliance certification for Part 2 is charged separately from Part 1. That is, there is a separate fee.

[jerstlouis]

Thanks @ghobona . So @pvretano my concern for simple Parts is back :)

[pvretano]

@jerstlouis so you are basing your concerns on economic considerations? I think that the Parts of common should reflect what the state of common practice is at the time the specifications are written. Of course, writing specification takes time so there would be some drift but generally the common practice is clear. For Part 2 of common, for example, the use of /collections is a well established practice. The use of bbox/datetime/limit at the /collections endpoint and the "collections of collections" concept are not common practice at the moment so I would not include them in Part 2.

[jerstlouis]

@pvretano I agree that in order to publish Part 2 sooner while reflecting common practice, offloading the Simple Query and Hierarchical Collections conformance classes to another part makes sense.

I think the economic considerations are valid for vendors (especially smaller companies) wishing to implement and be certified for as much as possible of the OGC API specifications, if several (tens? hundreds?) parts get published for single small conformance classes.

Also, "Simple Query" (bbox/datetime/limit on /collections), hierarchical collections, and records queries at /collections, would all work with many OGC API specifications, without them having to reference those. So I also wonder how this would be handled in CITE -- they would not have a corresponding Part to be certified for.

We're probably way off-topic here (sorry) and it's something to bring up with the CITE SC, but I feel that we should take these things into consideration when deciding where common conformance classes like Hierarchical Collections and Simple Query should end up.

Hence why I suggested OGC API - Common - Part X: Queryable and hierarchical collections, which would provide a CITE Test target (independent of the spec it is used with), and would also partially address economic concerns (e.g. grouping 3 conformance classes into 1 part to be certified for, and also significantly fewer than if Features, Maps, Tiles, Coverages, EDR each had to explicitly define a new part to support Hierarchical Collections, Simple Query, or Records Search @ /collections).

[cmheazel]

• Common Part 2 describes the collections and collection resources as well as the HTTP verbs and query parameters applicable to each. The Collections and Simple Query Conformance Classes are well within this scope.
• There is no difference between a queryable and non-queryable collection. All collections must meet the same requirements. What makes a Collection queryable is the support, or lack of support, for the query parameters (bbox, datetime, limit) by the API server.
• The Simple Query Conformance Class is meaningless without a Collections resource. To move it into a separate document simply makes things harder for the implementer.
• CITE tests conformance to a Conformance Class, not a to a standard. This will be made even more explicit as we move more into modular standards.
• See API-Coverages for an example of how Common could be used.

[cmheazel]

@pvretano So you are saying that we cannot filter the collections to be included in a /Collections response? That I cannot say "give me the collections that fall within this bounding box?" That IS common practice, especially for imagery.
Remember, an image is a coverage which is a collection.

[pvretano]

@cmheazel I am saying that should not be in Part 2. The "core" of the /collections endpoint should, more-or-less, match the functionality currently offered by features.

[jerstlouis]

@cmheazel I think it is super important to clearly distinguish between querying on "a" collection vs. querying on a "set of" collections.

The Simple Query Conformance class is useful to query the "set of" collections.

In this way, it serves a similar purpose to Records - Part 2 collections extension, and to some extent the proposed Hierarchical Collection as well, where querying a particular parent could only return that parent's immediate children to drill down to the collections of interest.

In terms of OGC API implementations so far, I agree with @pvretano in that I don't think bbox and dateTime at the /collections level has been widely implemented yet.

For EO imagery, that is an interesting topic, because on one hand it is useful to present the whole collection of images as a single coverage, but on the other it is useful to "discover" scenes of interest. For this I have been proposing the concept of a Scenes API expanding upon the Testbed-15 Images API concept, and this could be implemented either as hierarchical collections, or as a /scenes/ sub-resource of the collection (and either end-points could support the Simple Query parameters bbox, dateTime...).

[cmheazel]

@jerstlouis Most OGC API Implementations so far have been based on API-Features. Since Features does not support filtering on the /collections path, it is no surprise that it has not been implemented in OGC APIs. However, this is a very common feature of existing systems. Simple Features is simple, optional, and captures common existing capabilities. I'm convinced that it will be used. And if not, it does no harm.

[cmheazel]

@pvretano The functionality provided by API-Features addresses the needs of the Features community. That is not the universe. In my world the Simple Query functionality is ubiquitous. Part 2 is useless without it.

[jerstlouis]

@cmheazel

However, this is a very common feature of existing systems. Simple Features is simple, optional, and captures common existing capabilities. I'm convinced that it will be used. And if not, it does no harm.

I am in full agreement, but I would make the same argument about Hierarchical Collections :)

Part 2 is useless without it.

I would disagree with that though. Part 2 is already very useful with just the Core conformance class, in that clients and servers can implement this /collections and /collections/{collectionId} functionality that already works the same across Features, Coverages, Maps, Tiles and EDR.

But additional conformance classes like Simple Query, Hierarchical Collections and Records Queries (currently Records Part 2-collections extension) makes it more useful, especially for large number of collections.

Where those conformance classes should get defined should consider the overall implications for OGC in terms of drafting the standards, managing them, writing tests for them, the implementations getting certified for them, and procuring them, as discussed above.

[jerstlouis]

At the OGC API - Common session of the 127th Members Meeting in Singapore we briefly discussed this topic and there was no outspoken objection to draft and review an optional "Hierarchical collections" requirements class for Part 2 adding which would:

• Add a parent property to a collection object, which can reference a parent {collectionId}
• Add a parent parameter for GET /collections request which would limit the response to immediate children of the specified collection ID (a special none value would need to be determined to retrieve the root collections as opposed to the default full hierarchy when parent is omitted)

This would also replace capabilities that were specifically included in the 3D GeoVolumes spec ( opengeospatial/ogcapi-3d-geovolumes#5 and opengeospatial/ogcapi-3d-geovolumes#12 ).