Using a self-signed certificate with intake

[rsignell-usgs]

I'm a visiting scientist at an organization that uses a self-signed certificate.

col = intake.open_esm_datastore("https://cmip6-pds.s3.amazonaws.com/pangeo-cmip6.json",)

is failing in aiohttp/connector.py with

ClientConnectorError: Cannot connect to host cmip6-pds.s3.amazonaws.com:443 ssl:default [Connection reset by peer]

I'm able to set up a connection with

import ssl
import aiohttp

CA_BUNDLE="/etc/pki/tls/certs/ca-bundle.crt"
PEM_PUB="/home/la.signell/PA-RootCA-Cert-2023-Pub.pem"

ssl_ctx = ssl.create_default_context(cafile=CA_BUNDLE)
ssl_ctx.load_verify_locations(PEM_PUB)

We know this certificate works because we told Firefox to trust it and then it can open the catalog.

But how do we communicate this to Intake?

[martindurant]

You can apparently use the AWS_CA_BUNDLE environment variable to point to the crt, https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#environment-variable-configuration also available as ca_bundle= which I think goes in client_kwargs.

I'm not sure you need the PEM, I see no mention of it.

[martindurant]

Sorry, sorry - you are using the HTTP endpoint.
Apparently the aiohttp takes ssl= as the kwarg. intake-esm seems to take storage_options which I assume are passed straight to aiohttp, so maybe

col = intake.open_esm_datastore("https://cmip6-pds.s3.amazonaws.com/pangeo-cmip6.json", storage_options={"ssl": ssl_ctx})

[rsignell-usgs]

That gave me a type error -- but seems like it should be something like that. I'll raise the issue on the intake-esm repo, as perhaps @mgrover1 or others there have experienced something like this.

[mgrover1]

I have not seen this yet... but yeah! Go ahead and open an issue @rsignell-usgs