Look for older RHEL/Fedora /etc/pki/tls cert store location

[geofft]

RHEL 8 (supported until 2029) and below, including Fedora 33 and below, do not ship an /etc/ssl/cert.pem or a hashed /etc/ssl/cert/ directory, so our build of OpenSSL does not pick up platform certs. It would be nice if we patched our Linux build to look at /etc/pki/tls/cert.pem instead, if that file exists and /etc/ssl/cert.pem does not. Preferably this should be in OpenSSL, but if that's gnarly we can probably get away making this change in the _ssl stdlib module.

You can test this with the redhat/ubi8 or fedora:33 Docker images, and something like import urllib.request; urllib.request.urlopen("https://astral.sh").

See astral-sh/uv#16703 and https://bugzilla.redhat.com/show_bug.cgi?id=1053882 for more discussion. (I also saw something somewhere about p11-kit having an OpenSSL engine or provider plugin to load the certs from the proper place in the proper way, which we could explore, but seems much more fragile, and would probably also involve figuring out where and how to find the plugin.)

[jjhelmus]

This may be related to #259

[paveldikov]

This would be superb! Just out of curiosity, where is this sitting on the priority ladder? Would like to know if we should try to band-aid on our end, or plan to roll-forward if a fix is imminent?

Also worth mentioning that RHEL7 isn't entirely dead either, although that happens to use the same overridden truststore path. Just don't drop glibc-level compatibility and we should be golden.

[lorengordon]

I was just testing a python project where we are converting the standalone binary build to use pyapp with a custom distribution of python-build-standalone (instead of pyinstaller). Ran straight into this issue. Binary works fine on RHEL 9 systems, but errors with CERTIFICATE_VERIFY_FAILED messages on RHEL 8 systems. I'm brand new to python-build-standalone and creating custom distributions... Any tips on what I might tweak/patch in the distribution to get things working?

[geofft]

If it's an option for you, there are some simple workarounds:

• export SSL_CERT_FILE=/etc/pki/tls/cert.pem before starting Python
• os.environ['SSL_CERT_FILE'] = '/etc/pki/tls/cert.pem' inside Python before using SSL (maybe guarded with if not os.path.exists('/etc/ssl/cert.pem') or something to handle non-RHEL/Fedora machines too
• if you have admin access on the machines, create a symlink ln -s ../pki/tls/cert.pem /etc/ssl.

Re actually fixing it out of the box, to @paveldikov's question - I probably won't have time to look into it this week at least, @jjhelmus would you have time?

(And that's good to know about glibc compatibility, we were pondering raising it a bit but we can hold off....)

[jjhelmus]

I'll take a look at this later this week. I suspect that this will require patching OpenSSL, which is not something that should be done lightly and I'm hesitant to do at all. Perhaps this can be addressed within the Python _ssl module which would be more promising.

[lorengordon]

If it's an option for you, there are some simple workarounds:

• export SSL_CERT_FILE=/etc/pki/tls/cert.pem before starting Python
• os.environ['SSL_CERT_FILE'] = '/etc/pki/tls/cert.pem' inside Python before using SSL (maybe guarded with if not os.path.exists('/etc/ssl/cert.pem') or something to handle non-RHEL/Fedora machines too
• if you have admin access on the machines, create a symlink ln -s ../pki/tls/cert.pem /etc/ssl.

Thanks for the hints! I recalled that we already have a urllib wrapper, where we create the ssl context if necessary in the first place. Patched that as below, and all is working...

            context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)  # from `import ssl`

            # Workaround for old OpenSSL bug on Red Hat 8 systems
            # See: https://github.com/astral-sh/python-build-standalone/issues/858
            ssl_cert = Path("/etc/ssl/cert.pem")
            el8_cert = Path("/etc/pki/tls/cert.pem")
            if not ssl_cert.exists() and el8_cert.exists():
                context.load_verify_locations(cafile=str(el8_cert))