Copernicus Data Space Ecosystem #583
Comments
|
As far as I know, the new Copernicus Data Space Ecosystem uses very different data discovery and distribution mechanisms:
Not sure how the support for good old file download will look like. CreoDIAS Finder has that option and maybe that will stay around. Anyone know when Copernicus Open Access Hub (aka SciHub) will be decommissioned? A lot of the logic of Sentinelsat will become obsolete when we no longer need to jump through hoops to pull single files out of Copernicus' nose (e.g. the whole Long Term Archive access misery). In a way, it is weird that we needed a community-developed client library for properly making use of the most basic functionality of the Copernicus API, because their own clients were insufficient. Maybe there will be enough interest and someone willing to create a new client library that implements Sentinelsat's API and metadata format against the new official Copernicus services. I hope though that the new services will lend themselves better to standard tooling and more people will be able to use data-proximate compute. |
|
The new discovery (and download) APIs feature OData and OpenSearch interfaces. https://documentation.dataspace.copernicus.eu/#/APIs The data returned by both follows the same schema and is valid GeoJSON. Download links point at https://zipper.creodias.eu - product contents are gathered from object storage and zipped before shipping. {
"type": "FeatureCollection",
"properties": {
"id": "83a3744f-d43e-5ead-be06-0e40533d3a1a",
"totalResults": 416859,
"exactCount": true,
"startIndex": 1,
"itemsPerPage": 1,
"query": {
"originalFilters": {
"collection": "Sentinel2",
"sortParam": "startDate",
"maxRecords": "1"
},
"appliedFilters": {
"collection": "Sentinel2",
"sortParam": "startDate",
"maxRecords": "1"
},
"processingTime": 4.939621241
},
"links": [
{
"rel": "self",
"type": "application/json",
"title": "self",
"href": "http://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel2/search.json?startDate=2021-07-01&completionDate=2021-07-31&sortParam=startDate&maxRecords=1"
},
{
"rel": "search",
"type": "application/opensearchdescription+xml",
"title": "OpenSearch Description Document",
"href": "http://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel2/describe.xml"
},
{
"rel": "next",
"type": "application/json",
"title": "next",
"href": "http://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel2/search.json?page=2&startDate=2021-07-01&completionDate=2021-07-31&sortParam=startDate&maxRecords=1"
},
{
"rel": "last",
"type": "application/json",
"title": "last",
"href": "http://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel2/search.json?page=416859&startDate=2021-07-01&completionDate=2021-07-31&sortParam=startDate&maxRecords=1"
}
]
},
"features": [
{
"type": "Feature",
"id": "033d1996-57a5-5787-b76c-11ebda56774a",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-126.62152,
76.56115660434
],
[
-126.6219,
76.561079263695
],
[
-127.09917,
76.459685917284
],
[
-127.569916,
76.357466825105
],
[
-128.0344,
76.254413536922
],
[
-128.49226,
76.150491401071
],
[
-128.9438,
76.045833401294
],
[
-129.00075,
76.032329963613
],
[
-129.00078,
76.581028747004
],
[
-126.62152,
76.56115660434
]
]
]
},
"properties": {
"collection": "SENTINEL-2",
"status": "ONLINE",
"license": {
"licenseId": "unlicensed",
"hasToBeSigned": "never",
"grantedCountries": null,
"grantedOrganizationCountries": null,
"grantedFlags": null,
"viewService": "public",
"signatureQuota": -1,
"description": {
"shortName": "No license"
}
},
"productIdentifier": "[/eodata/Sentinel-2/MSI/L1C/2021/06/30/S2B_MSIL1C_20210630T220059_N0301_R072_T09XWE_20210630T222928.SAFE](https://catalogue.dataspace.copernicus.eu/eodata/Sentinel-2/MSI/L1C/2021/06/30/S2B_MSIL1C_20210630T220059_N0301_R072_T09XWE_20210630T222928.SAFE)",
"parentIdentifier": null,
"title": "S2B_MSIL1C_20210630T220059_N0301_R072_T09XWE_20210630T222928.SAFE",
"description": null,
"organisationName": "ESA",
"startDate": "2021-06-30T22:00:59.024Z",
"completionDate": "2021-06-30T22:00:59.024Z",
"productType": "S2MSI1C",
"processingLevel": "S2MSI1C",
"platform": "S2B",
"instrument": "MSI",
"resolution": 60,
"sensorMode": "INS-NOBS",
"orbitNumber": 22549,
"quicklook": null,
"thumbnail": "https://finder.creodias.eu/files/Sentinel-2/MSI/L1C/2021/06/30/S2B_MSIL1C_20210630T220059_N0301_R072_T09XWE_20210630T222928.SAFE/S2B_MSIL1C_20210630T220059_N0301_R072_T09XWE_20210630T222928-ql.jpg",
"updated": "2021-07-01T00:34:30.516Z",
"published": "2021-07-01T00:34:30.516Z",
"snowCover": 0,
"cloudCover": 11.0645,
"gmlgeometry": "<gml:Polygon srsName=\"EPSG:4326\"><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>-126.62152,76.56115660434 -126.6219,76.561079263695 -127.09917,76.459685917284 -127.569916,76.357466825105 -128.0344,76.254413536922 -128.49226,76.150491401071 -128.9438,76.045833401294 -129.00075,76.032329963613 -129.00078,76.581028747004 -126.62152,76.56115660434</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon>",
"centroid": {
"type": "Point",
"coordinates": [
-128.21454713163658,
76.39319815863165
]
},
"orbitDirection": "ASCENDING",
"timeliness": null,
"relativeOrbitNumber": 72,
"processingBaseline": 3.01,
"missionTakeId": "GS2B_20210630T220059_022549_N03.01",
"services": {
"download": {
"url": "https://zipper.creodias.eu/download/033d1996-57a5-5787-b76c-11ebda56774a",
"mimeType": "application/octet-stream",
"size": 0
}
},
"links": [
{
"rel": "self",
"type": "application/json",
"title": "GeoJSON link for 033d1996-57a5-5787-b76c-11ebda56774a",
"href": "http://catalogue.dataspace.copernicus.eu/resto/collections/SENTINEL-2/033d1996-57a5-5787-b76c-11ebda56774a.json"
}
]
}
}
]
} |
|
Because you asked when the Copernicus Open Access Hub (aka SciHub) will be decommissioned, that is planned for June 2023: https://scihub.copernicus.eu/news/News01146 |
|
Has anyone started or is interested in getting started on a Python package that uses the new Copernicus Data Space Ecosystem OData API? I would like to contribute :) |
|
Here is a minimalist lib for the CreoDIAS Finder OpenSearch API: https://github.com/DHI-GRAS/creodias-finder - very quickly hacked together once, but maybe you can salvage some parts. OData seems to be recommended, though. |
|
Good morning, i'm a user of Sentinelsat tool that helped me many times in searching and downloading, in an automatic way, a lot of sentinel data. I really hope that i will be able to use it on the new ecosystem, when the actual hub will be disabled. I'm hot a developer and so i hope that someone will help people like me to reach this goal. Thanks to everyone |
|
I have not used it myself, but also |
|
Thank you @fwfichtner , I need to test that, it looks like a real powerful tool. |
On the basis of the documentation provided, multiple APIs are available for diving the data (OSearch, OData and STAC) and even an S3 archive for downloading. One possibility would be converting |
|
I have managed to do a query with standard parameters such a collection, aoi intersect etc, as in the example below, start_date = "2022-06-01"
end_date = "2022-06-10"
data_collection = "SENTINEL-1"
product_type = 'SLC'
aoi = "POLYGON((4.220581 50.958859,4.521264 50.953236,4.545977 50.906064,4.541858 50.802029,4.489685 50.763825,4.23843 50.767734,4.192435 50.806369,4.189689 50.907363,4.220581 50.958859))'"
json = requests.get(
f"https://catalogue.dataspace.copernicus.eu/odata/v1/Products?$filter=Collection/Name eq '{data_collection}' and OData.CSC.Intersects(area=geography'SRID=4326;{aoi}) and Attributes/OData.CSC.StringAttribute/any(att:att/Name eq 'productType' and att/OData.CSC.StringAttribute/Value eq '{product_type}') and ContentDate/Start gt {start_date}T00:00:00.000Z and ContentDate/Start lt {end_date}T00:00:00.000Z"
).json()
pd.DataFrame.from_dict(json["value"]).head()
but then I do not know how to use the s3path to download the data. Anybody can help on that? |
|
From the OData REST API you should now get an access token and use the Ids to download. This is not too different from the old API of the Data Hub. https://documentation.dataspace.copernicus.eu/APIs/OData.html#product-download So for instance: On the basis of current doc pages, it seems the STAC API is still incomplete at this stage and the most practical thing is implementing the OData update for the new system. About the S3, I'm not sure the catalog is routing an S3 protocol even for non DIAS computing resources. |
|
Hi @mdelgadoblasco , I implemented the CDSE based on OData for one of my scripts and according to my needs: Now you have the options of access and refresh tokens. |
|
Thank you very much! I tried to use/adapt your code for S1 SLC data but it simply does not seem to work. I get always 301 code. Anybody can bring some light why this 301 code happens to S1 SLC products? Maybe are not still available? |
301 is a permanent move, probably the result is routed elsewhere and you need to follow the new URL to get the result, something that curl does automagically. I see that the Python code above does not manage that case, instead. |
|
@fpl I thought the following block of code (line 373) was taking care of the 301 error: Seems like I need to check and see if the problem is from my or their code. |
|
I also used it but it never leaves the while loop.... let me know if you manage solve it. |
|
Yes @mdelgadoblasco , you are right, I tried with your input example and in fact it stays too much time inside the loop and at the end it generates the wrong .zip file. If you remove the .zip extension you will see that saves the error code only. I tested with Is there an option in requests for |
|
Ouh! perfect! I will use the provide code above . ;) |
j08lue
added
scihub-replacement
|
Hi all! How can I solve that? |
|
In the last weeks, I've been migrating my other code to CDSE, I don't see any special network issue, I'm currently able to download in a multi-threaded queue even 20 files in a few minutes with 10 workers or more. What you are pointing out seems to be a network issue, but I'm not sure if you have exhausted your quota and now finding timeouts for bandwidth reduction. For sure, I can't note anything like that. The nominal quota should be at least 6TB/month with 4 threads, but that is not enforced apparently now. EDIT: starting from the beginning of October, it seems the 4 stream limit is actually enforced. Sad, but at least all products seem online. |
|
I summarize here some information about the CDSE services for future use, at this point in time: the OSearch API has some inconsistencies to be considered in the workflow. There is a general limitation of 200,000 products to deal with a query. While page 1 of a big query can be loaded in any case, successive calls with |
|
Hi all, I see there is the possibility that the utility provided by sentinelsat can be no longer available due to the changes in Copernicus services. The main reason I always used sentinelsat (thank you to this nice community!) was to deal with queries of many products (more than one) and dowload it all without taking care too much of all the mess related with OData coding... I read the current info from ESA, but it seems there is no such a thing in their API to download more than one product... am I right? Any suggestion to replace my service based on sentinelsat with something else (Python based) for the specific purpose of just download of multiple products? Thank you very much.... |
I'm not sure what you mean, but of course, OData can be used to download up to 4 products concurrently with the same credentials. There are quite a few differences between the old and the new service protocols, so sentinelsat needs a few changes here and there, something I could complete in a few time starting from the next week. In the mean time the old service was dismantled yesterday. |
|
Thank you @fpl for your reply. What I mean is this, with sentinelsat I could search images with SentinelAPI.query() using some keywords and/or input data like geometries and dates. The method returned a dataframe with a list of products, that I could give to download_all() to download all of them, easily. Now, reading ESA docs (https://dataspace.copernicus.eu/news/2023-9-28-accessing-sentinel-mission-data-new-copernicus-data-space-ecosystem-apis) I see there is code to download one product at a time. In my understanding sentinelsat was a very useful utility to avoid coding all this one-by-one download scripts, to get the work of downloading many products easily done.. hope this clarifies! |
|
Hi @massimozanetti , you can still do that with CDSE with some adjustments. For example, currently I am using a clumsy (I must admit) function that I developed to search and download from CDSE using Python, my output is a text file with something like this: Results for Sentinel-2L1C using:
Now is start working on adapting sentinelsat on free times, any help you need I will be glad to help :) |
|
If it's any help, feel free to look into how we created our functions that interact with the CDSE. |
|
Hi! I created a CDSE download script. Its currently in early stages, so there's lost missing but i plan on keeping up development on it. https://github.com/SDFIdk/CDSE_downloader I hope it can do some good. |
Thanks, I'm currently rigorously working with a tactical approach by simply trying to get sentinelsat tests working. That is largely suboptimal by a good design PoV, but it seems the most effective mode of supporting the new protocols. I agree with others: rewriting from scratch is probably the fastest way of having something working at large, but the evil is in details, and I had to works several weeks in my other project about CDSE to have a decent result finally. |
|
I reported to CDSE service network issues with the new services, specifically in downloading multi-GBs images. This is annoying, really. Even the forum is full of people who have problems.
|
|
CDSETool is now a published python package and can now query and download reliably. |
|
does anyone know what is correct way to filter S1 query by th |
The OData attributes can be expanded by appending
|
|
this one should work fine, but see my clumsy solution starting on line 160: indeed, NO DOCUMENTATION for parameters and those changed!!! CDSE did NOT do good job here (at least they could have supported sentinelsat development) |
Thank you @espiritocz, I have found my error: wrong one: the first CSC attribute must be String and the second is Integer. Thank you @fpl for the info, means a lot. |
|
Is there any documentation on the meaning of the fields the |
I guess you are talking about some of the OData results. You should ignore both fields if empty: I'm sad for you. In a perfect world, one would expect they should be used to get the archive size and its (MD5 and others) checksums. In our real world, they are 0 and an empty list. That said, I'm afraid you have to use an OData by Name AND COLLECTION search instead, e.g.: I'm quite sure this is a current bug in the CDSE server software. Note the use of SENTINEL-2 as collection name, which is different and not compatible with Sentinel2, despite documentation. |
|
Hi @fpl - happy to relocate this discussion to somewhere else if it is more appropriate. I need to download Sentinel-5P files for multiple years. This will total > 10 TB. To do this, I gathered links through OData. The links look like this: http://catalogue.dataspace.copernicus.eu/odata/v1/Products(bd285023-8ff4-5574-87f5-de0a64bd240c)/$value My goal is then to use my Copernicus maximum of 4 concurrent connections to download this data as fast as possible. I have tried this with the simple bash script below, but the download takes much too long. In particular, every few files, it seems to be limited for a few minutes before picking up again. At the current rate, it would take weeks to download the files. Have you found any effective bulk download methods? Thanks for any help you can provide! |
|
Seems like a good case for https://gis.stackexchange.com/, actually. |
hi. I was doing the same mistake - you should not be regenerating ACCESS_TOKEN as their number is limited (!) You have to generate one ACCESS_TOKEN and then use the REFRESH TOKEN instead (that is not well documented, indeed..). |
|
Hi, I had built yet another quick version of API to substitute the former Sentinelsat API since the CopernicusHUB decommissioning, then only now to find this issue and many other options developed for downloading from Copernicus Data Space. This version uses OData API and imports each of data collection as a subclass. It currently implements only Copernicus mission datasets per my projects' needs, but could be easily extended for the few other datasets available on dataspace. Though I guess it isn't a smart way of making this package cause those external datasets may grow more in the future. Would be happy to hear anyone finding it useful or willing to contribute and improve. |
Hi @fpl, do you know if there is an equivalent (machine readable) description for other APIs? |
|
On Tue, Apr 16, 2024 at 03:34:40AM -0700, Antonio Valentino wrote:
> [...] For instance, CDSE now provides an XML description for each collection:
>
> ```
> https://catalogue.dataspace.copernicus.eu/resto/api/collections/<collection>/describe.xml
> ```
>
> with relevant information, such as validation patterns and a list of parameters.
Hi @fpl, do you know if there is an equivalent (machine readable) description for other APIs?
Not at my knowledge. Even, a pair of those XML descriptions are partially
malformed and still are, months after opening a ticket about
that :-/ Probably you could ask on the forum or opening a ticket to have
an answer about that, I guess a similar metainformation doc should be
available even for other APIs.
…--
Francesco P. Lovergine | ***@***.***
|
Hello,
Here is a news from ESA which announce the interruption of service of Copernicus Data Hub at September 2023.
https://scihub.copernicus.eu/news/News01146
Is it schedule to sentinelsat to switch to the new service Data Space Ecosystem ?
Thanks for the anwser.
Best
Nicolas
The text was updated successfully, but these errors were encountered: