Remove expired Let's Encrypt (DST) Root certificate #162
Conversation
this certificate expired on "Sep 30 14:01:15 2021 GMT" and keeping it in the trusted store confuses OpenSSL 1.0.2 (like still present on EL7) and results in failed validations, even if there are other valid trust paths. see [1] for more details. [1] https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
|
This package is a direct repackaging of Mozilla's trust store. We do not make our own changes, has this cert been removed from there? |
|
Not yet. No. https://bugzilla.mozilla.org/show_bug.cgi?id=1733560 (and probably https://bugzilla.mozilla.org/show_bug.cgi?id=1733003, but nobody wants to wait till December) |
|
@dwt I believe this is your problem |
|
Great to see this going forward, if I can help test this, I'll gladly spend some time. |
|
I've added DST to the Makefile, so that Can obviously be reverted once Mozilla actually drops that cert from their store. |
|
Concretely, we do not editorialise in this package: we repackage what Mozilla has. |
|
@Lukasa I'd like to reference the thoughts I shared in #142 (comment) . More specifically, it is now my understanding, that including this certificate makes it so that python 2 using urllib3 / requests (which includes pip) on at least centos 7 boxes are unable to validate servers that have LetsEncrypt certificates. To me that certainly is a big deal, and I hope that this fact can spark some discussions on how to best deal with the problem here. To reiterate some thoughts: Mozilla controls both the list of certificates as well as the client library (and it's version) using said list, so they don't have to deal with these kind of problems but can deal with them on the software side. We/you don't have that luxury as this software also runs on the py2/centos7 combo with the unfortunately old OpenSSL version. So until a workaround is available on another layer for these systems this is really nasty. Just for posterity: my current workaround is declaring an environment variable outside or inside of python: $ REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem http https://dependencies.yeepa.de# Workaround for bug in python2.7 where requests doesn't like to validate
# the letsencrypt certificate we have deployed on dependencies.yeepa.de
# Probably because the certificates in the certifi package are missing somehting.
import os
os.environ['REQUESTS_CA_BUNDLE'] = '/etc/pki/tls/cert.pem'Before calls to requests / urllib3 happens. |
|
Mozilla/NSS doesn't have a problem with having the expired DST cert still in the store, but OpenSSL (and thus users of |
|
In the meantime, manually removing it from the installed .pem file in production does the trick... (I never said that). |
|
I understand the concerns, but certifi is not attempting to solve this problem. It's a redistribution of the Mozilla trust store: nothing more. We are not running a root program and we are not setting trust policy for the wider Python ecosystem. I agree that there are problems with certificate validation in Python programs, but this is a problem for Python to solve, and where you get your Python from your OS vendor then it is a problem for them to solve. The solutions are well understood and are not excessively complex, but there has thus far been insufficient funding to do the engineering work to address them. Ultimately, certifi is working as designed. You'll need to take this up with your OS vendors and any downstreams using certifi in a way you're unhappy about. |
|
@Lukasa What are the options you can fathom then? Some things I can think of:
|
|
Most OS vendors patch I think the main problem is when users want to run something in a virtual env, and just want to do |
This package doesn't support Python 2. It dropped support for it, so I'm not sure why we'd make a special wheel for it.
This wouldn't work either.
Cory was talking about the fact that the system Python you're depending on uses a version of OpenSSL where this is in fact an issue. If you're using 2.7 (and it's still supported) on a system that provides it for you, you need to raise a stink with the folks who maintain that distribution as to the fact that this whole thing is a problem on their supported software. |
sigmavirus24
changed the title
Following the expiration of the DST Root CA X3 certificate, we need the certifi package to remove the certificate from its list of CA authorities [1]. However, it will have to wait until the cert is removed from the Mozilla root store [2][3]. This patch works around the issue by using the REQUESTS_CA_BUNDLE environment variable, and making it point to the system-provided CA bundle. [1] - certifi/python-certifi#162 [2] - https://bugzilla.mozilla.org/show_bug.cgi?id=1733560 [3] - https://bugzilla.mozilla.org/show_bug.cgi?id=1733003 Change-Id: Ieace8ba8b59b52730bbd391daaa909eb42d05036
|
Hm, this of course happens in a virtualenv, so the vendor provided certifi package is not used in this case. I don't think I can fault the vendor in this case. |
|
The vendor is absolutely still on the hook because they gave you your OpenSSL and your Python. If you'd like to build a new OpenSSL and a Python that links it, that will fix your problem. |
|
Just a vote here for the project to remove the offending certificate (as I've done on my local install) and push an update sooner, rather than later, given that there are a lot of larger institutions that I'm forced to deal with that are still running Python 2.7 on RHEL, and it's like pulling teeth getting newer packages installed (as I'd need to do, assuming that a more recent OpenSSL is available on that distribution release). This outstanding issue breaks the "requests" library pretty severely in those places where we're using Let's Encrypt certificates. |
|
As others have stated, this affects venvs, not necessarily distributions directly. For example on CentOS 7 the system python 2.7 and requests package continue to work fine, however venvs which pull in requests and certifi directly from pypi run into trouble. On CentOS 7 the system has which provides a ca-cert bundle that contains all the same ca-certs the latest certifi packages does, except for the expired one. So my pragmatic workaround is to simply update all my python 2.7 venvs on those nodes containing certifi for some reason via (this is roughly equivalent to simply removing the expired cert from the bundle, although in my institution some additional local certs are added to the trust bundle) The issue I have with this is that it is not clear at all from the error message why there is a sudden problem with certificate verification in services which have operated fine in some venv for extended periods of time. It took a substantial amount of time to determine the cause for these sudden verification failures. I hope this information helps other folks. |
|
3 maintainers have already said that we won't alter the bundle generated by Mozilla arbitrarily. As such I'm locking this because this conversation isn't productive |
this certificate expired on "Sep 30 14:01:15 2021 GMT" and keeping it in
the trusted store confuses OpenSSL 1.0.2 (like still present on EL7) and
results in failed validations, even if there are other valid trust
paths. see [1] for more details.
[1] https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/