Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query by tile: unfound Sentinel-2 product #556

Closed
remi-braun opened this issue Nov 25, 2022 · 1 comment · Fixed by #713
Closed

Query by tile: unfound Sentinel-2 product #556

remi-braun opened this issue Nov 25, 2022 · 1 comment · Fixed by #713
Labels
bug Something isn't working

Comments

@remi-braun
Copy link
Contributor

remi-braun commented Nov 25, 2022

Describe the bug
When querying a Sentinel-2 product by tile (with the provided shapefile), the result may be empty for partial products (edge of swath) if the tile centroid is outside the S2 footprint.

Code To Reproduce

import os
import tempfile

from eodag import EODataAccessGateway, setup_logging
from eodag.config import override_config_from_file

from eodownload.dir_utils import get_data_directory

OUTPUT = r"D:\_EODAG\OUTPUT\"

os.environ["EODAG__EARTH_SEARCH__DOWNLOAD__EXTRACT"] = "false"
os.environ["EODAG__EARTH_SEARCH__DOWNLOAD__OUTPUTS_PREFIX"] = OUTPUT
os.environ[
    "EODAG__EARTH_SEARCH__AUTH__CREDENTIALS__AWS_ACCESS_KEY_ID"
] = "xxx"
os.environ[
    "EODAG__EARTH_SEARCH__AUTH__CREDENTIALS__AWS_SECRET_ACCESS_KEY"
] = "xxx"
os.environ["EODAG__EARTH_SEARCH__AUTH__CREDENTIALS__AWS_PROFILE"] = "image-download"

if __name__ == "__main__":
    setup_logging(3)  # DEBUG level

    with tempfile.TemporaryDirectory() as tmp:

        locations_yaml_content = """
            shapefiles:
              - name: s2_tiles_centroids
                path: {}
                attr: index
            """.format(
            get_data_directory() / "s2_tiles_centroids.shp"
        )

        custom_loc = os.path.join(tmp, "custom_locations.yml")
        with open(custom_loc, "w") as f_yml:
            f_yml.write(locations_yaml_content.strip())

        # Create gateway
        dag = EODataAccessGateway(locations_conf_path=custom_loc)
        dag.set_preferred_provider("earth_search")

        # Query all
        products = dag.search_all(
            start="2022-05-07",
            end="2022-05-08",
            **{
                "productType": "S2_MSI_L2A",
                "locations": dict(s2_tiles_centroids="36NUG"),
            }
        )

        # Download all
        dag.download_all(products)

Output

2022-11-25 11:46:40,486 eodag.config                     [INFO    ] (config           ) Loading user configuration from: C:\Users\rbraun\.config\eodag\eodag.yml
2022-11-25 11:46:40,631 eodag.core                       [INFO    ] (core             ) usgs: provider needing auth for search has been pruned because no crendentials could be found
2022-11-25 11:46:40,631 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-11-25 11:46:40,755 eodag.core                       [DEBUG   ] (core             ) Opening product types index in C:\Users\rbraun\.config\eodag\.index
2022-11-25 11:46:40,760 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from C:\Users\rbraun\AppData\Local\Temp\tmpygpootyf\custom_locations.yml
2022-11-25 11:46:40,760 eodag.config                     [INFO    ] (config           ) Loading user configuration from: D:\_EODOWNLOAD\eodownload\eodownload\data\sertit_conf_template.yml
2022-11-25 11:46:40,776 eodag.config                     [INFO    ] (config           ) usgs: unknown provider found in user conf, trying to use provided configuration
2022-11-25 11:46:40,776 eodag.config                     [INFO    ] (config           ) aws_eos: unknown provider found in user conf, trying to use provided configuration
2022-11-25 11:46:41,335 eodag.core                       [DEBUG   ] (core             ) Searching for all the products with provider earth_search and a maximum of 500 items per page.
2022-11-25 11:46:42,121 eodag.core                       [INFO    ] (core             ) Searching product type 'S2_MSI_L2A' on provider: earth_search
2022-11-25 11:46:42,122 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: StacSearch
2022-11-25 11:46:42,122 eodag.core                       [INFO    ] (core             ) Iterate search over multiple pages: page #1
2022-11-25 11:46:42,122 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Mapping eodag product type to provider product type
2022-11-25 11:46:42,122 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S2_MSI_L2A
2022-11-25 11:46:42,203 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-11-25 11:46:42,203 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-11-25 11:46:42,203 eodag.plugins.search.qssearch    [INFO    ] (qssearch         ) Sending search request: https://earth-search.aws.element84.com/v0/search
2022-11-25 11:46:42,203 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Query parameters: {'collections': ['sentinel-s2-l2a'], 'datetime': '2022-05-07T00:00:00.000Z/2022-05-08T00:00:00.000Z', 'intersects': {'type': 'Point', 'coordinates': [31.695828581223143, 1.3127651504842646]}, 'limit': 500, 'page': 1}
2022-11-25 11:46:43,107 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page Query-object could not be collected
2022-11-25 11:46:43,122 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page merge could not be collected
2022-11-25 11:46:43,122 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 0 plugin results to eodag product representation
2022-11-25 11:46:43,122 eodag.core                       [DEBUG   ] (core             ) Iterate over pages: last products found on page 0
2022-11-25 11:46:43,122 eodag.core                       [INFO    ] (core             ) Found 0 result(s) on provider 'earth_search'
2022-11-25 11:46:43,122 eodag.core                       [INFO    ] (core             ) Empty search result, nothing to be downloaded !

However a product does exist here:
2022-11-25_12h04_40

Environment:

  • Python version: 3.9.13
  • EODAG version: version 2.6.2.post1
@remi-braun remi-braun added the bug Something isn't working label Nov 25, 2022
@sbrunato
Copy link
Collaborator

Hello @remi-braun , thanks very much for pointing out this issue.
Searching tiles using the centroids like in the tutorial does not seem fully reliable for all providers.
We should test this example on other providers and mention in the tutorials the one having this issue.

Also related to #255 , to natively query using tiles when the provider supports it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants