Basic example SSLCertVerificationError

[ZZMitch]

Hello,

I am trying to run through the basic example (https://stackstac.readthedocs.io/en/latest/basic.html) but running into an SSLCertVerificationError on the following cell:

import pystac_client URL = "https://earth-search.aws.element84.com/v1" catalog = pystac_client.Client.open(URL)

The final error: APIError: HTTPSConnectionPool(host='earth-search.aws.element84.com', port=443): Max retries exceeded with url: /v1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

I did some googling and this may be related to my VPN (I am a government employee working from home). But I am not certain, so putting this here just in case...

Thanks!

[TomAugspurger]

stac-utils/pystac-client#383 and stac-utils/pystac-client#88 might be relevant.

…

On Thu, Oct 26, 2023 at 11:11 AM Mitchell Bonney ***@***.***> wrote: Hello, I am trying to run through the basic example ( https://stackstac.readthedocs.io/en/latest/basic.html) but running into an SSLCertVerificationError on the following cell: import pystac_client URL = "https://earth-search.aws.element84.com/v1" catalog = pystac_client.Client.open(URL) The final error: APIError: HTTPSConnectionPool(host=' earth-search.aws.element84.com', port=443): Max retries exceeded with url: /v1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) I did some googling and this may be related to my VPN (I am a government employee working from home). But I am not certain, so putting this here just in case... Thanks! — Reply to this email directly, view it on GitHub <#228> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIW47PXQSCKI5QT6NELYBKDUPBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44SQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJTGQ2TAMBUG43DBAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDCOJWGM4TAMJSGY2KO5DSNFTWOZLSUZRXEZLBORSQ> . You are receiving this email because you are subscribed to this thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[ZZMitch]

Thanks @TomAugspurger. That pystac-client issue 383 sounds exactly like mine. It looks like I need to find a custom certificate file...

Now to find out what that is and if we have one!

[ZZMitch]

So yes, I can confirm that installing a certificate and then using a verify command to link my code with the the location of the cer file allows me to access STAC through the VPN.

I have been able to complete most of the basic example until the point when I actually need to load the images into memory:

import dask.diagnostics with dask.diagnostics.ProgressBar(): data = aoi.compute()

At this point I get another certification related error:
RuntimeError: Error opening 'https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/13/S/DV/2020/5/S2A_13SDV_20200503_1_L2A/B03.tif': RasterioIOError('CURL error: schannel: CertGetCertificateChain trust error CERT_TRUST_IS_UNTRUSTED_ROOT')

Will keep investigating!

[gjoseph92]

@ZZMitch did you get anywhere with this? My guess is that, if you can figure out what rasterio.Env parameters you need to get a basic rasterio.open to work locally, you can use that to pass in a custom gdal_env to stackstac. Happy to help with that part once you have something working, but I'd start with how to set a custom root certificate with GDAL.

[Kirill888]

few years back this was common problem with rasterio on Ubuntu systems, with work-around being setting this env var
export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt or something along these lines, but I'd first try updating certifi package first, it's probably too old in your setup.

[ZZMitch]

Hey, sorry to leave this thread hanging! Yes, I was able to solve this issue with the help of @sgillies. See rasterio/rasterio#2954 for details.

It was indeed an env parameter: rasterio.Env(GDAL_HTTP_UNSAFESSL = 'YES'). There may be a more secure option, but I have not had need to explore deeper yet.

I will note that it seems to take longer to load the imagery into memory than I might expect (i.e., based on the time shown in the example and running this on a non-corporate network in the past), but I am not sure if that is related to this. My comment from the above link on this:

Another issue is that it takes longer than expected (though I am happy it is working at least!). It takes 40-50 seconds to load into memory, with no progress for the first 8-10 seconds. According to the stackstac basic example, it should take 10-15 seconds. Again, maybe some GDAL configuration options may help here.

I am not sure if implementing within stackstac.stack, rather than at the time of using compute would make any difference in this regard.

That said, you can feel free to close this issue now if needed.

[gjoseph92]

It sounds like @ZZMitch is on a system where IT has set up "SSL Visibility": they're decrypting all SSL traffic and re-encrypting it with their own certificate, which the root certificates bundled with certifi (rightly) reject as invalid, so you have to use a custom root certificate they provide. This was my experience working in a government agency a while back at least!

It makes sense that setting GDAL_HTTP_UNSAFESSL would work around that. If you wanted to still verify SSL, just with the custom certificate, then CURL_CA_BUNDLE sounds like the thing to use—thanks @Kirill888.

In that case, it would look like (sorry this is so awkward, xref #133):

root_ca = "path/to/certificate/provided/by/IT"
gdal_env = stackstac.DEFAULT_GDAL_ENV.updated(always=dict(CURL_CA_BUNDLE=root_ca))

stackstac.stack(..., gdal_env=gdal_env)

it seems to take longer to load the imagery into memory than I might expect

There are a lot of variables in this. If IT is decrypting your SSL traffic, they're probably also scanning it in some way; that alone can add a lot of latency. (They're probably scanning all the GeoTIFFs for viruses, etc.)

I'd be surprised if GDAL_HTTP_UNSAFESSL slowed things down significantly on its own. If possible, you might try running the same code side by side on a corporate and non-corporate machine on the same network (say at home). If it's slow in both cases, then that would be good to look into.

I'll close this for now though, but feel free to re-open if something comes up. Thanks for the update!

[sgillies]

@gjoseph92 thank you! I learned a lot here.

[ZZMitch]

Hey @gjoseph92, a follow-up question for you...

I have been interested in having the Dask dashboard (i.e., https://github.com/dask/dask-labextension) to assist me with selecting optimum chunk sizes and better monitoring of imagery computations. However, it seems like my network is also problematic for this.

For example, if I add the Client:

from dask.distributed import Client
client = Client() # security = True
client

When I later want to compute my imagery:

import rasterio

with rasterio.Env(GDAL_HTTP_UNSAFESSL = 'YES') as env:
    data = aoi.compute()

I get this error:
RuntimeError: Error opening 'https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/13/S/DV/2020/5/S2A_13SDV_20200503_0_L2A/B02.tif': RasterioIOError("'/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/13/S/DV/2020/5/S2A_13SDV_20200503_0_L2A/B02.tif' does not exist in the file system, and is not recognized as a supported dataset name.")

If I comment out the Client cell, everything works as intended. On the other hand, if I use the Dask dashboard for monitoring local data processing, everything works fine. So it some combination of the cloud data access and Dask that is triggering this. I looked at some Client options (https://docs.dask.org/en/stable/deploying-python.html?highlight=client#client) and the only thing that seems relevant is security. Setting that the True did not change anything, and I am not sure what the "security object" they want is.

This may be a better question for Dask, but thought I would ask here since you seem knowledgeable on this topic.

[gjoseph92]

@ZZMitch I think this is a typical issue of forwarding authentication to the workers in your dask cluster. I don't know how you're authenticating with AWS, but I'd bet there's something you're doing in your Python session to set it up (maybe setting something in rasterio.Env). When you launch a dask cluster, the workers are separate Python processes, and may not inherit those settings from your current process.

stackstac doesn't have a built in way of forwarding authentication, and as I said here I'm not sure it's something I'd want to get into, since it's a much more general issue that comes up all the time with dask: #154 (reply in thread).

Also FWIW, see #133 — when you do this, the rasterio env you just set is not passed on to the workers:

with rasterio.Env(GDAL_HTTP_UNSAFESSL = 'YES') as env:
    data = aoi.compute()

If you want to adjust the rasterio environment, you'll need to do something like gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(GDAL_HTTP_UNSAFESSL ='YES'))

https://jacobtomlinson.dev/posts/2022/how-to-set-environment-variables-on-your-dask-workers/ may also be helpful.

[ZZMitch]

Thanks! I am not authenticating with AWS at all, I more just wanted to make sure I was making best use of my local hardware to do computations. I am very new with Dask, so I may need to do some more learning to better understand how it works.

[ZZMitch]

@ZZMitch I think this is a typical issue of forwarding authentication to the workers in your dask cluster. I don't know how you're authenticating with AWS, but I'd bet there's something you're doing in your Python session to set it up (maybe setting something in rasterio.Env). When you launch a dask cluster, the workers are separate Python processes, and may not inherit those settings from your current process.

stackstac doesn't have a built in way of forwarding authentication, and as I said here I'm not sure it's something I'd want to get into, since it's a much more general issue that comes up all the time with dask: #154 (reply in thread).

Also FWIW, see #133 — when you do this, the rasterio env you just set is not passed on to the workers:

with rasterio.Env(GDAL_HTTP_UNSAFESSL = 'YES') as env:
    data = aoi.compute()

If you want to adjust the rasterio environment, you'll need to do something like gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(GDAL_HTTP_UNSAFESSL ='YES'))

https://jacobtomlinson.dev/posts/2022/how-to-set-environment-variables-on-your-dask-workers/ may also be helpful.

@gjoseph92 this is totally accurate and I am annoyed at myself at not making this change back in December!

See my issue in the Dask discussion forum... The information you provided here was the fix, but I had forgotten about it until today. Interestingly I was able to work with a dask LocalCluster with original way of setting GDAL_HTTP_UNSAFESSL, but it only seemed to function in some situations. Passing the required GDAL settings with your method on stackstac.stack() is clearly the robust way to do it though.