consider looking in /etc/pki/ca-trust/extracted for CA certificates?

[tlambert03]

Hello,

Recently helped a colleague debug an issue with SSL certs. Here was the code we were testing:

import tensorstore as ts
store = ts.open(
    {
        "driver": "n5",
        "kvstore": {
            "driver": "s3",
            "bucket": "janelia-cosem-datasets",
            "path": "jrc_hela-3/jrc_hela-3.n5/labels/er-mem_pred/s4",
        },
    }
).result()

and he got:

ValueError: UNKNOWN: Error opening "n5" driver: 
Error reading "jrc_hela-3/jrc_hela-3.n5/labels/er-mem_pred/s4/attributes.json":
CURL error[77] Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations: 
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
[source locations='tensorstore/internal/http/curl_transport.cc:456\ntensorstore/internal/cache/kvs_backed_cache.h:208\ntensorstore/driver/driver.cc:117']
[tensorstore_spec='{\"context\":{\"aws_credentials\":{\"filename\":\"\",\"metadata_endpoint\":\"\",\"profile\":\"\"},\"cache_pool\":{},\"data_copy_concurrency\":{},\"s3_request_concurrency\":{},\"s3_request_retries\":{}},\"driver\":\"n5\",\"kvstore\":{\"aws_region\":\"us-east-1\",\"bucket\":\"janelia-cosem-datasets\",\"driver\":\"s3\",\"path\":\"jrc_hela-3/jrc_hela-3.n5/labels/er-mem_pred/s4/\"}}']

He's on RedHat:

LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description:    Red Hat Enterprise Linux release 8.6 (Ootpa)
Release:        8.6
Codename:       Ootpa

and we eventually found that, unlike the Debian standard of /etc/ssl/certs/ca-certificates.crt, the standard location for CA Certs is /etc/pki/ca-trust/extracted, so setting export TENSORSTORE_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem fixed our issues. However, that does need to be done before tensorstore is imported, making it a bit harder for my package (which uses tensorstore) to guarantee that a redhat user won't run into this issue.

Would it be possible for tensorstore to perform a check for this file? (and also, please correct me if I'm wrong about the "standardness" of that location)

[laramiel]

We might be able to rely on boringssl resolving the certificate path by removing the #define here https://github.com/google/tensorstore/blob/fc0ba0b6b17620695d86139b7ec367de9666b41b/third_party/se_curl/curl.BUILD.bazel#L204C20-L204C34

Is your SSL_CERT_DIR environment variable set?

[laramiel]

Or we could do something similar to gRPC: https://github.com/grpc/grpc/blob/23adb994cfbb91a66d0d5a52a4cc07a2a42c2d53/src/core/lib/security/security_connector/load_system_roots_supported.cc#L51

[tlambert03]

Awesome, thanks so much @laramiel