Bulk download

[heidivanparys]

From https://www.w3.org/TR/dwbp/#BulkAccess :

Best Practice 17: Provide bulk download
Enable consumers to retrieve the full dataset with a single request.

[...]

Possible Approach to Implementation
Depending on the nature of the data and consumer needs, possible approaches could include the following:

• [...]
• Hosting an API that includes the ability to retrieve a bulk download in addition to dynamic queries. This approach is useful for capturing a complete snapshot of dynamic data.
• [...]

Has a mechanism for doing this with the OGC API Features been considered?

[pvretano]

@heidivanparys Do you man request all features from all collections is a single request?
Yes, I think there has been some discussion about a /items path in OGC API Common -- basically all items available at the endpoint (collection-less). Also, there was some work done in previous testbed on ASYNCHRONOUS requests ... because I would image such a request might take longer than "usual" time to process.
There was also work done on specifying multiple collections in a single request and I suppose one could enumerate all collections or we could specify a special token like "*" as a wildcard to mean all collections.
So, work has been done, but nothing has been put into core so far.

[heidivanparys]

Ok, thanks.

I was actually thinking of the use case where one or more of the collections contain a lot of features, a number higher than the maximum limit of the server. Paging is then required and this means that the dataset cannot be downloaded with one request. A solution could be to provide bulk download of a static file (to use the wording of https://www.w3.org/TR/dwbp/#dataAccess : data that is pre-processed server side). This static file may even may be located somewhere else. Which probably would make it another distribution of the dataset.

So my question is maybe rather this one: Has a mechanism for linking to another distribution of the same dataset with the OGC API Features been considered? Or would the correct way be to go to the metadata of the dataset and find any other distributions from there?

[jvanulde]

If we unpack this there is a lot of things to consider... not so say that they haven't been considered. For example, a (self describing) collection resource can provide links to related resources (e.g. features) which may or may not provide the same representation (e.g. GML). It's an edge case that perhaps we will never see but none-the-less it's not inconceivable. So, if you dereference a collection and it provides representations (e.g. GeoPackage) is the assumption that you get all features that are related to that collection? That may be difficult for the reason stated above. I suppose you could lean on WPS to do the heavy lifting.

[cholmes]

I'm definitely interested in a 'bulk' endpoint, doing exactly what you are saying @heidivanparys - provide the data without making clients page through. I think for many it would need to be requested async. And it's interesting to consider the 'static' use case, where basically the bulk endpoint doesn't accept any 'query'. But you could see providing shapefile, geopackage and gml versions of the full dataset. And ideally the 'collection' has links to them that specify the full mimetype and that 'query' is not accepted. More dynamic servers could accept query/filter for creation of subsets / different formats at the bulk endpoint, and would do that async.

Don't think it needs to be in core, but would definitely be a great extension. Indeed it's one that can help bridge to more traditional GIS - make your query through the web, but return a filtered shapefile or geopackage that you can load up with ease.

So I'd see a lightweight extension that is just a 'link' on the collection object with a specific resource type that enables you to say 'this set of items is alternately available as one download with this mimetype'.

And then an additional extension that gives the server paths / requests to enable a user to turn any 'query' into async output without paging.

[jampukka]

I've used #152 for this

[jvanulde]

If the filtering wasn't a requirement you could just specify a link with the "enclosure" relation signifying that the resource is potentially large and requires specialized handling (e.g. GIS).

[pvretano]

Is filtering really a requirement for a "bulk" request? As per @heidivanparys original post, the request is for the "full" dataset which implies no filtering ... no? In that case the rel="enclosure" approach seem like a good solution and does not even require any changes to the specification, not even the implementation of issue #152. The href of the link would be opaque and the server could mint whatever URL it likes for the purpose of fetching the entire collection.
If however, filtering and fetching multiple collections at once is a requirement then, as @jvanulde pointed out, the link might not be the best approach.
So, it filtering and fetching multiple collections at the same time a requirement for a "bulk" fetch capability?

[cportele]

Has a mechanism for linking to another distribution of the same dataset with the OGC API Features been considered? Or would the correct way be to go to the metadata of the dataset and find any other distributions from there?

Based on the definition in DCAT, a bulk download would be another distribution of the same dataset next to the API and this is why such downloads have not been discussed in the document. From DCAT:

Represents a specific available form of a dataset. ... Examples of distributions include a downloadable CSV file, an API or an RSS feed.

So one way of finding the other distributions including bulk downloads would be to go to the dataset metadata and inspect the distribution records. But including an enclosure link in an OGC API response is perfectly fine, too, and does not require a change to the schemas as Peter said. We could add an enclosure link in an example to show how to do it.

There is the recommendation that enclosure links should include information about the length of the referenced document. I assume we keep the link object for now as it is and do not add a length member in the Core. In the XML it is already there due to the use of atom:link and in the JSON one can add additional members.

[heidivanparys]

So, it filtering and fetching multiple collections at the same time a requirement for a "bulk" fetch capability?

Filtering: no, fetching multiple collections at the same time: yes (as "a bulk download would be another distribution of the same dataset"), fetching one complete collection at the same time: could be nice.

We could add an enclosure link in an example to show how to do it.

Would this then look like the example below?

{
  "links": [
    { "href": "http://data.example.org/collections.json",
      "rel": "self", "type": "application/json", "title": "this document" },
    { "href": "http://data.example.org/collections.html",
      "rel": "alternate", "type": "text/html", "title": "this document as HTML" },
    { "href": "http://schemas.example.org/1.0/buildings.xsd",
      "rel": "describedBy", "type": "application/xml", "title": "GML application schema for Acme Corporation building data" },
    { "href": "http://download.example.org/buildings.gpkg",
      "rel": "enclosure", "type": "application/geopackage+sqlite3", "title": "Bulk download (GeoPackage)" }
  ],
  "collections": [ ... ]
}

[cportele]

Would this then look like the example below?

Yes. Maybe with an additional "length": 472546 member.

[pvretano]

@heidivanparys fetching one complete collection at the same time could be done exactly the same way. That is, in the links section of the collection metadata you include a link rel="enclosure" that fetches the entire collection in some representation (such as GeoPackage)
GET /collections/buildings
{
"id": "buildings",
"title": "Buildings",
"description": "Buildings in the city of Bonn.",
"extent": { ... },
},
"links": [
..., { "href": "http://data.example.org/collections/buildings/items",
"rel": "enclosure", "type": "application/geopackage+sqlite3",
"length": 127390854,
"title": "Bulk Download of Buildings Collection in GeoPackage Format" }, ...
]
}

[cportele]

Discussion 17 June: Agreement to include this, but the open question is where. In the Guide, in the Core document in an example (but are we overloading the document?), in an extension? Discuss and decide in Leuven.