Change PEPS provider to ESA scihub copernicus #36

jeremy-cesbio · 2022-10-05T09:30:47Z

Feature request
PEPS does not distribute Sentinel-2 images since the beginning of 2020. These images are distributed by the ESA Copernicus scihub. Changing or adding that provider would allow users to get the S2_MSI_L2A data directly from the source that distributes it.

sbrunato · 2022-10-05T14:14:30Z

Hello @jeremy-cesbio , this provider is already available through eodag-sentinelsat plugin.

This is in a separate package because it uses sentinelsat package which is under GPL3 license and is incompatible with Apache2

jeremy-cesbio · 2022-10-05T15:00:43Z

Hello @sbrunato, thanks for your reply ! I managed to install the module without issues.

However there seems to be an issue, I have a ConnectTimeoutError when looking for products on SciHub, I can't figure out why (my credentials are correct), do you have a little time to help me out ?

sbrunato · 2022-10-06T13:00:58Z

can you please post a code snippet to reproduce the error, and also the output you get with maximum verbosity ?
For the snippet, for example:

from eodag import EODataAccessGateway, setup_logging

setup_logging(3) # 0: nothing, 1: only progress bars, 2: INFO, 3: DEBUG

dag = EODataAccessGateway()
dag.set_preferred_provider("scihub")

search_results, _ = dag.search(
    productType="S2_MSI_L1C",
    start="2022-08-01",
    end="2028-08-31",
    geom={"lonmin": 1, "latmin": 43, "lonmax": 2, "latmax": 44}
)

jeremy-cesbio · 2022-10-06T14:15:34Z

Here is what I'm trying to run :

from eodag import EODataAccessGateway, setup_logging

setup_logging(3) # 0: nothing, 1: only progress bars, 2: INFO, 3: DEBUG

dag = EODataAccessGateway()
dag.set_preferred_provider("scihub")

search_criteria = {
    "productType": "S2_MSI_L2A",
    "start": "2021-05-01",
    "end": "2021-09-01",
    "geom": {"lonmin": 1, "latmin": 43, "lonmax": 2, "latmax": 44}
}

all_products = dag.search_all(**search_criteria)

Here is the eodag config file for scihub :

scihub:
    priority: 2  # Must be higher than the other providers' priorities
    api:
        extract: 
        outputs_prefix: /path/to/my/download/directory/
        credentials:
            username: my_username
            password: my_password

Here is the result I obtain :

2022-10-06 14:07:39,350 eodag.config                     [INFO    ] (config           ) Loading user configuration from: /home/ad/auclaij/.config/eodag/eodag.yml
2022-10-06 14:07:39,463 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-10-06 14:07:39,567 eodag.core                       [DEBUG   ] (core             ) Opening product types index in /home/ad/auclaij/.config/eodag/.index
2022-10-06 14:07:39,578 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from /home/ad/auclaij/.config/eodag/locations.yml
2022-10-06 14:07:40,374 eodag.core                       [DEBUG   ] (core             ) Searching for all the products with provider scihub and a maximum of 100 items per page.
2022-10-06 14:07:40,374 eodag.core                       [INFO    ] (core             ) Searching product type 'S2_MSI_L2A' on provider: scihub
2022-10-06 14:07:40,375 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: SentinelsatAPI
2022-10-06 14:07:40,375 eodag.core                       [INFO    ] (core             ) Iterate search over multiple pages: page #1
2022-10-06 14:07:40,376 eodag.plugins.apis.sentinelsat   [DEBUG   ] (eodag_sentinelsat) Initializing Sentinelsat API
2022-10-06 14:07:40,377 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Mapping eodag product type to provider product type
2022-10-06 14:07:40,378 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S2_MSI_L2A
2022-10-06 14:07:40,382 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-10-06 14:07:40,383 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-10-06 14:07:40,384 eodag.plugins.apis.sentinelsat   [INFO    ] (eodag_sentinelsat) Sending query request with `sentinelsat`
2022-10-06 14:07:40,385 sentinelsat.SentinelAPI          [DEBUG   ] (sentinel         ) Running query: order_by=None, limit=100, offset=0, query=beginPosition:["2021-05-01T00:00:00Z" TO "2021-09-01T00:00:00Z"] platformname:"Sentinel-2" producttype:"S2MSI2A" footprint:"Intersects(POLYGON ((1.0000 43.0000, 1.0000 44.0000, 2.0000 44.0000, 2.0000 43.0000, 1.0000 43.0000)))"
2022-10-06 14:07:40,386 sentinelsat.SentinelAPI          [DEBUG   ] (sentinel         ) Sub-query: offset=0, limit=100
2022-10-06 14:09:47,663 eodag.core                       [INFO    ] (core             ) No result from provider 'scihub' due to an error during search. Raise verbosity of log messages for details
Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?0dec7728-1697-40f2-adb1-d6d97ed993de)
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:

File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/util/connection.py:95, in create_connection(address, timeout, source_address, socket_options)
     94 if err is not None:
---> 95     raise err
     97 raise socket.error("getaddrinfo returns an empty list")

File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84     sock.bind(source_address)
---> 85 sock.connect(sa)
     86 return sock

TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError                       Traceback (most recent call last)
File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/connectionpool.py:703, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
...
--> 553         raise ConnectTimeout(e, request=request)
    555 if isinstance(e.reason, ResponseError):
    556     raise RetryError(e, request=request)

ConnectTimeout: HTTPSConnectionPool(host='apihub.copernicus.eu', port=443): Max retries exceeded with url: /apihub/search?format=json&rows=100&start=0&q=beginPosition%3A%5B%222021-05-01T00%3A00%3A00Z%22+TO+%222021-09-01T00%3A00%3A00Z%22%5D+platformname%3A%22Sentinel-2%22+producttype%3A%22S2MSI2A%22+footprint%3A%22Intersects%28POLYGON+%28%281.0000+43.0000%2C+1.0000+44.0000%2C+2.0000+44.0000%2C+2.0000+43.0000%2C+1.0000+43.0000%29%29%29%22 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2ac351e3a050>, 'Connection to apihub.copernicus.eu timed out. (connect timeout=None)'))

sbrunato · 2022-10-07T07:33:45Z

I don't manage to reproduce your error... Could you also post eodag and sentinelsat versions ?

import eodag; print(eodag.__version__)
import sentinelsat; print(sentinelsat.__version__)

jeremy-cesbio · 2022-10-07T07:45:38Z

Here is what I obtain :

import eodag; print(eodag.__version__)
import sentinelsat; print(sentinelsat.__version__)

2.5.2
1.1.1

sbrunato · 2022-10-07T09:44:39Z

I see in your logs that you are running the code in the CNES environment. Have you correctly set proxy settings ?
Please try executing this:

import requests
r = requests.get("https://apihub.copernicus.eu")
print(r.status_code)

jeremy-cesbio · 2022-10-07T10:11:56Z

I have a technical proxy that work correctly on a terminal. But it seems my proxy isn't working when using a Jupyter Notebook. However I can get search results with providers peps and theia.

import requests
r = requests.get("https://apihub.copernicus.eu")
print(r.status_code)

And here is the result :

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/connection.py:174, in HTTPConnection._new_conn(self)
    173 try:
--> 174     conn = connection.create_connection(
    175         (self._dns_host, self.port), self.timeout, **extra_kw
    176     )
    178 except SocketTimeout:

File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/util/connection.py:95, in create_connection(address, timeout, source_address, socket_options)
     94 if err is not None:
---> 95     raise err
     97 raise socket.error("getaddrinfo returns an empty list")

File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84     sock.bind(source_address)
---> 85 sock.connect(sa)
     86 return sock

TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError                       Traceback (most recent call last)
File /work/scratch/auclaij/envs_dirs/modspa_env/lib/python3.10/site-packages/urllib3/connectionpool.py:703, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
...
--> 553         raise ConnectTimeout(e, request=request)
    555 if isinstance(e.reason, ResponseError):
    556     raise RetryError(e, request=request)

ConnectTimeout: HTTPSConnectionPool(host='apihub.copernicus.eu', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x2b3d27277dc0>, 'Connection to apihub.copernicus.eu timed out. (connect timeout=None)'))

How do I fix that ?

Thanks again for your time !

sbrunato · 2022-10-07T12:02:23Z

However I can get search results with providers peps and theia.

You do not need proxy to access https://peps.cnes.fr or https://theia.cnes.fr. Check that http_proxy and https_proxy are correctly set in your .bashrc or equivalent, restart your session after changes, and if you still have issues please contact CNES HPC support. Good luck!

jeremy-cesbio · 2022-10-07T12:09:21Z

Thank your for your time !

Running python scripts in commande line does not cause any issue, it seems it's the Jupyter Notebooks that causes problems.

jeremy-cesbio added the enhancement New feature or request label Oct 5, 2022

sbrunato transferred this issue from CS-SI/eodag Oct 6, 2022

jeremy-cesbio closed this as completed Oct 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change PEPS provider to ESA scihub copernicus #36

Change PEPS provider to ESA scihub copernicus #36

jeremy-cesbio commented Oct 5, 2022

sbrunato commented Oct 5, 2022

jeremy-cesbio commented Oct 5, 2022

sbrunato commented Oct 6, 2022

jeremy-cesbio commented Oct 6, 2022

sbrunato commented Oct 7, 2022

jeremy-cesbio commented Oct 7, 2022

sbrunato commented Oct 7, 2022 •

edited

Loading

jeremy-cesbio commented Oct 7, 2022 •

edited

Loading

sbrunato commented Oct 7, 2022

jeremy-cesbio commented Oct 7, 2022

Change PEPS provider to ESA scihub copernicus #36

Change PEPS provider to ESA scihub copernicus #36

Comments

jeremy-cesbio commented Oct 5, 2022

sbrunato commented Oct 5, 2022

jeremy-cesbio commented Oct 5, 2022

sbrunato commented Oct 6, 2022

jeremy-cesbio commented Oct 6, 2022

sbrunato commented Oct 7, 2022

jeremy-cesbio commented Oct 7, 2022

sbrunato commented Oct 7, 2022 • edited Loading

jeremy-cesbio commented Oct 7, 2022 • edited Loading

sbrunato commented Oct 7, 2022

jeremy-cesbio commented Oct 7, 2022

sbrunato commented Oct 7, 2022 •

edited

Loading

jeremy-cesbio commented Oct 7, 2022 •

edited

Loading