Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Stop trying to locate system trust stores #3416
We started trying to locate the system trust stores, because downstream was patching out our bundled copies anyways and it would provide a smoother experience when people upgraded their pip inside of their system.
However, if we just use OpenSSL's CAFile then we're broken on systems like Debian which currently ship a broken CAFile configuration. If we just use OpenSSL's CAPath then we're broken on systems like CentOS and Fedora that currently are shipping a broken OpenSSL CAPath.
So basically, none of the major distributions seem to be capable of shipping an OpenSSL that isn't broken, so we're going back to relying on our own CA bundle exclusively.
added a commit
this pull request
Jan 21, 2016
Jan 21, 2016
1 check passed
I don't know if this is the appropriate venue, but this change sits extremely poorly with me (and with lots of other security-minded system administrators, I'd bet). A lot of environments rely on audited and trimmed system SSL CA paths, and unless people are reading the changelog, Pip is now going to be running arbitrary code trusting a vendorized and un-reviewed CA list from the
Have you tried reaching out to Debian and Red Hat about their respective issues? There are hundreds of packages on our Debian and Red Hat based systems successfully using the system CA trust store, so I don't think it's quite as dire as the linked thread suggests.
@Roguelazer On behalf of the requests team I'd like to thank you for notifying us about our problems with SSL handling: as I'm sure you know, silently patching projects rather than reporting problems with them works great. Would you mind clarifying what you believe are our flaws, as it applies to pip in this case?
I don't really want to go into my issues with requests; in fact, I'm sorry I brought it up here. Let me make this discussion more productive. In order to be able to upgrade to pip8 (and thus not get the annoying nag message) and not have to change a huge number of build scripts for first and third party software that use pip to set the relevant requests environment variable, I would really like one of the following things done w.r.t. this change:
Lacking that, I guess I'll be setting the
Unfortunately I agree with @Lukasa - the issues you're concerned about should be addressed upstream in requests and not in pip. We switched back to using requests' default behaviour because working around the various distro issues was unproductive for us.
If "system trust store by default" is a sensible approach, then you should propose it to the requests team. If they agree, then pip will pick it up when we upgrade our vendored copy (and others benefit as well). If they don't agree, then at least the dialogue is occurring between the right parties.
I would argue that the sensitivity of pip (which runs arbitrary code on every request) is significantly higher than requests, which is an HTTP library for people who don't want to understand HTTP, and that its settings demand much higher sensitivity.
I've also had that discussion with the requests team several years ago and have no particular desire to be in any way, shape, or form involved with the requests project.
@Roguelazer a search of requests issues show no issues opened by you, mentioning you, or commented on by you. If you've participated in a discussion on requests then either you've changed your username, or you've deleted everything which means that we cannot address your concerns. Since PyPA members are asking you to address the "bad SSL handling and for which every company I've worked for or with ends up overriding the SSL CA settings", would you at least enumerate some of your concerns here with examples?
To be clear, with requests API in the form that it is in today pip cannot use the system trust store effectively, because it cannot pass both a CA directory and a CA file. It would need to, because it is not possible for pip to analytically determine whether a given path is a valid source of CAs without using OpenSSL extensively to try to load certs, which is a bizarre and onerous requirement.
Requests could in principle make that API change, but there is a question about whether we should do this properly more generally. As a team we are extremely reluctant to introduce different behaviours on different platforms, or to provide Linux users with a different logic than we provide Windows or OS X users.
However, despite your pretty negative opinion of us, we're not here to make your life difficult, and in fact we've been discussing this for a little while now at requests/requests#2966. However, for us to fix this in a way that is universally helpful and secure is not trivial.
I don't believe we can reasonably use the system trust store in a cross platform way given the APIs that are available to us right now.
Some background: There are two "kinds" of trust paths that OpenSSL understands, there is
Previously we attempted to detect the system CA Bundle by simply enumerating a list of known file locations looking for one that existed. However that was causing issues in edge cases because there are a number of systems in the wild that have a stale (or possibly empty) ca file in one of those locations which happened to be the wrong location for that particular system (for example, the location for that OS was
So next we attempted to use the OpenSSL API to locate a CAFile, however it quickly was discovered that some distros have that API configured incorrectly and the
Our next attempt was to switch to using just
Our next attempt was to switch to trying to utilize
At this point I gave up and just went back to what we originally did, which was only trust our bundled copy unless certifi was installed, in which case it will use that instead.
A few other points that are important here:
All of this whole mess makes it very difficult to correctly utilize the trust store when we don't know ahead of time what OS we're going to be running on and, even assuming just systems that use OpenSSL, if they have a trust store available to them, installed by default, or in what ways their OpenSSL is going to be misconfigured. We attempted to do it, but it turns out that it takes a lot more effort to do it in a way that isn't going to add a lot of false errors (and false errors are very bad, they train users to ignore certificate validation errors).
Due to all of the above, I felt it was in the best interest of all of our users to return to the one method that we know will always work (because the file is bundled and is always there) and which we know the method used to update and generate the bundle are actually secure and accurate (unlike some OS vendors) and which does not include unaudited CAs by default (unlike some OS vendors). I feel this is an acceptable solution since the CA certificates are hardly the only security sensitive code here, and we're just as likely to need to patch a security sensitive bug as we are to need to do a quick update of the CA bundle and any of our users will want to handle both scenarios (made easier by the update nag that pip 6+ does).
If someone comes up with a good way to handle all of the edge cases from above I'm certainly open up to looking at a patch, but I'm wary of trying to do it because it's proven very difficult to do correctly thus far, and the error case tends to be "make a release, wait an hour and get people yelling at me that FooBar Linux can no longer use pip". I trust requests (and the team behind it) that, if they start enabling this to happen, that it will happen in a way that is both secure and user friendly and we'll of course pick that up automatically when we update requests and at that point we wouldn't disable it or anything.