Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Download only selected bands/files on cop_dataspace #1018

Open
Rapsodia86 opened this issue Feb 8, 2024 · 4 comments · May be fixed by #1231
Open

Download only selected bands/files on cop_dataspace #1018

Rapsodia86 opened this issue Feb 8, 2024 · 4 comments · May be fixed by #1231
Assignees
@dalpasso
Labels
enhancement New feature or request

Comments

@Rapsodia86
Copy link

Hello,
I would like to ask if there is a chance at some point to extend some functionality for specific providers. Here, that would be Copernicus Dataspace.
Some datasets like e.g., Sentinel-1 or -2 can be pretty heavy, while a user may only need just a few bands (#910).
When querying thru oData e.g. "https://catalogue.dataspace.copernicus.eu/odata/v1" instead of downloading a whole zipped *.SAFE file, a user can download manifest XML only:
url = f"{catalogue_odata_url}/Products({product_identifier})/Nodes({product_name})/Nodes(MTD_MSIL1C.xml)/$value"

After taking band names from XML, a user can download files (bands) of interest. The whole approach is in https://github.com/eu-cdse/notebook-samples/blob/main/geo/odata_basics.ipynb (although there is a typo in the search (eu-cdse/notebook-samples#49).

Would that be something to consider?

Thanks!
@Rapsodia86 Rapsodia86 added the enhancement New feature or request label Feb 8, 2024
@sbrunato
Copy link
Collaborator

sbrunato commented Feb 9, 2024 •
edited
Loading

Hello @Rapsodia86 and thanks for this feature request and the associated inputs!

Single assets download is already implemented on the develop branch and will be shipped in the incoming v2.12.0.

You will be able to download individual assets from the new creodias_s3 provider or also STAC providers like astraea_eod, earth_search, planetary_computer, ....

Not yet with cop_dataspace, but we will plan to make it available using your inputs!

@sbrunato sbrunato changed the title Download only selected bands/files Download only selected bands/files on cop_dataspace Feb 13, 2024
@sbrunato
Copy link
Collaborator

sbrunato commented May 3, 2024

For this feature, OData should be used for creodias and cop_dataspace instead of OpenSearch

@sbrunato sbrunato mentioned this issue May 21, 2024
Closed
@sbrunato
Copy link
Collaborator

sbrunato commented May 29, 2024 •
edited
Loading

For instance, assets can be found through https://download.dataspace.copernicus.eu/odata/v1/Products(7fb094ad-ce1d-5e6e-91ed-7eeacb51e617)/Nodes
Then, following links recursively, we can find bands here https://download.dataspace.copernicus.eu/odata/v1/Products(7fb094ad-ce1d-5e6e-91ed-7eeacb51e617)/Nodes(S2A_MSIL1C_20200801T105631_N0209_R094_T30TYP_20200801T113256.SAFE)/Nodes(GRANULE)/Nodes(L1C_T30TYP_A026689_20200801T110247)/Nodes(IMG_DATA)/Nodes
Download links will be like https://download.dataspace.copernicus.eu/odata/v1/Products(7fb094ad-ce1d-5e6e-91ed-7eeacb51e617)/Nodes(S2A_MSIL1C_20200801T105631_N0209_R094_T30TYP_20200801T113256.SAFE)/Nodes(GRANULE)/Nodes(L1C_T30TYP_A026689_20200801T110247)/Nodes(IMG_DATA)/Nodes(T30TYP_20200801T105631_B01.jp2)/$value

See documentation here

@dalpasso dalpasso linked a pull request Jun 21, 2024 that will close this issue
search: add new plugin CopDataSpaceSearch #1231
Draft
@dalpasso
Copy link
Collaborator

dalpasso commented Jul 11, 2024 •
edited
Loading

Walking through the node tree of the products to discover the assets requires a high number of HTTP request (~10 request per product), leading to a high execution time. As alternative solution, the manifest file can be used to get all the assets at once.

The file name of the manifest is different among different Sentinel products:

  • for Sentinel-1 and Sentinel-2 the file is manifest.safe
  • for Sentinel-3 the file is xfdumanifest.xml
  • no manifest file for Sentinel-5

A post on the Copernicus Data Space forum has been created to have a clarification on the different behaviors: https://forum.dataspace.copernicus.eu/t/location-of-the-manifest-of-sentinel-products/933/1

@sbrunato sbrunato mentioned this issue Aug 16, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dalpasso dalpasso

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

search: add new plugin CopDataSpaceSearch CS-SI/eodag
3 participants
@Rapsodia86 @sbrunato @dalpasso