New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop trying to locate system trust stores #3416

Merged
merged 1 commit into from Jan 21, 2016

Conversation

Projects
None yet
6 participants
@dstufft
Member

dstufft commented Jan 21, 2016

We started trying to locate the system trust stores, because downstream was patching out our bundled copies anyways and it would provide a smoother experience when people upgraded their pip inside of their system.

However, if we just use OpenSSL's CAFile then we're broken on systems like Debian which currently ship a broken CAFile configuration. If we just use OpenSSL's CAPath then we're broken on systems like CentOS and Fedora that currently are shipping a broken OpenSSL CAPath.

So basically, none of the major distributions seem to be capable of shipping an OpenSSL that isn't broken, so we're going back to relying on our own CA bundle exclusively.

Fixes #3415

Review on Reviewable

Stop trying to locate system trust stores
We started trying to locate the system trust stores, because
downstream was patching out our bundled copies anyways and it would
provide a smoother experience when people upgraded their pip inside
of their system.

However, if we just use OpenSSL's CAFile then we're broken on systems
like Debian which currently ship a broken CAFile configuration. If
we just use OpenSSL's CAPath then we're broken on systems like CentOS
and Fedora that currently are shipping a broken OpenSSL CAPath.

So basically, none of the major distributions seem to be capable of
shipping an OpenSSL that isn't broken, so we're going back to relying
on our own CA bundle exclusively.

dstufft added a commit that referenced this pull request Jan 21, 2016

Merge pull request #3416 from dstufft/the-system-is-bad
Stop trying to locate system trust stores

@dstufft dstufft merged commit de6e4b5 into pypa:develop Jan 21, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@dstufft dstufft deleted the dstufft:the-system-is-bad branch Jan 21, 2016

@Roguelazer

This comment has been minimized.

Show comment
Hide comment
@Roguelazer

Roguelazer Jan 28, 2016

I don't know if this is the appropriate venue, but this change sits extremely poorly with me (and with lots of other security-minded system administrators, I'd bet). A lot of environments rely on audited and trimmed system SSL CA paths, and unless people are reading the changelog, Pip is now going to be running arbitrary code trusting a vendorized and un-reviewed CA list from the requests package (which is notorious for bad SSL handling and for which every company I've worked for or with ends up overriding the SSL CA settings).

Have you tried reaching out to Debian and Red Hat about their respective issues? There are hundreds of packages on our Debian and Red Hat based systems successfully using the system CA trust store, so I don't think it's quite as dire as the linked thread suggests.

Roguelazer commented Jan 28, 2016

I don't know if this is the appropriate venue, but this change sits extremely poorly with me (and with lots of other security-minded system administrators, I'd bet). A lot of environments rely on audited and trimmed system SSL CA paths, and unless people are reading the changelog, Pip is now going to be running arbitrary code trusting a vendorized and un-reviewed CA list from the requests package (which is notorious for bad SSL handling and for which every company I've worked for or with ends up overriding the SSL CA settings).

Have you tried reaching out to Debian and Red Hat about their respective issues? There are hundreds of packages on our Debian and Red Hat based systems successfully using the system CA trust store, so I don't think it's quite as dire as the linked thread suggests.

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jan 28, 2016

Contributor

@Roguelazer On behalf of the requests team I'd like to thank you for notifying us about our problems with SSL handling: as I'm sure you know, silently patching projects rather than reporting problems with them works great. Would you mind clarifying what you believe are our flaws, as it applies to pip in this case?

Contributor

Lukasa commented Jan 28, 2016

@Roguelazer On behalf of the requests team I'd like to thank you for notifying us about our problems with SSL handling: as I'm sure you know, silently patching projects rather than reporting problems with them works great. Would you mind clarifying what you believe are our flaws, as it applies to pip in this case?

@Roguelazer

This comment has been minimized.

Show comment
Hide comment
@Roguelazer

Roguelazer Jan 28, 2016

I don't really want to go into my issues with requests; in fact, I'm sorry I brought it up here. Let me make this discussion more productive. In order to be able to upgrade to pip8 (and thus not get the annoying nag message) and not have to change a huge number of build scripts for first and third party software that use pip to set the relevant requests environment variable, I would really like one of the following things done w.r.t. this change:

  1. Go back to using the system trust store by default
  2. Add a setting to re-enable the old behavior of using the configured Python trust root (something like --cert-trust-system)

Lacking that, I guess I'll be setting the cert parameter in /etc/pip.conf. This works today, although I don't like it quite as much as (2) because then I have to denormalize the path to my SSL CA list into pip.conf

Roguelazer commented Jan 28, 2016

I don't really want to go into my issues with requests; in fact, I'm sorry I brought it up here. Let me make this discussion more productive. In order to be able to upgrade to pip8 (and thus not get the annoying nag message) and not have to change a huge number of build scripts for first and third party software that use pip to set the relevant requests environment variable, I would really like one of the following things done w.r.t. this change:

  1. Go back to using the system trust store by default
  2. Add a setting to re-enable the old behavior of using the configured Python trust root (something like --cert-trust-system)

Lacking that, I guess I'll be setting the cert parameter in /etc/pip.conf. This works today, although I don't like it quite as much as (2) because then I have to denormalize the path to my SSL CA list into pip.conf

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 28, 2016

Member

Unfortunately I agree with @Lukasa - the issues you're concerned about should be addressed upstream in requests and not in pip. We switched back to using requests' default behaviour because working around the various distro issues was unproductive for us.

If "system trust store by default" is a sensible approach, then you should propose it to the requests team. If they agree, then pip will pick it up when we upgrade our vendored copy (and others benefit as well). If they don't agree, then at least the dialogue is occurring between the right parties.

Member

pfmoore commented Jan 28, 2016

Unfortunately I agree with @Lukasa - the issues you're concerned about should be addressed upstream in requests and not in pip. We switched back to using requests' default behaviour because working around the various distro issues was unproductive for us.

If "system trust store by default" is a sensible approach, then you should propose it to the requests team. If they agree, then pip will pick it up when we upgrade our vendored copy (and others benefit as well). If they don't agree, then at least the dialogue is occurring between the right parties.

@Roguelazer

This comment has been minimized.

Show comment
Hide comment
@Roguelazer

Roguelazer Jan 28, 2016

I would argue that the sensitivity of pip (which runs arbitrary code on every request) is significantly higher than requests, which is an HTTP library for people who don't want to understand HTTP, and that its settings demand much higher sensitivity.

I've also had that discussion with the requests team several years ago and have no particular desire to be in any way, shape, or form involved with the requests project.

Roguelazer commented Jan 28, 2016

I would argue that the sensitivity of pip (which runs arbitrary code on every request) is significantly higher than requests, which is an HTTP library for people who don't want to understand HTTP, and that its settings demand much higher sensitivity.

I've also had that discussion with the requests team several years ago and have no particular desire to be in any way, shape, or form involved with the requests project.

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Jan 28, 2016

Member

@Roguelazer a search of requests issues show no issues opened by you, mentioning you, or commented on by you. If you've participated in a discussion on requests then either you've changed your username, or you've deleted everything which means that we cannot address your concerns. Since PyPA members are asking you to address the "bad SSL handling and for which every company I've worked for or with ends up overriding the SSL CA settings", would you at least enumerate some of your concerns here with examples?

Member

sigmavirus24 commented Jan 28, 2016

@Roguelazer a search of requests issues show no issues opened by you, mentioning you, or commented on by you. If you've participated in a discussion on requests then either you've changed your username, or you've deleted everything which means that we cannot address your concerns. Since PyPA members are asking you to address the "bad SSL handling and for which every company I've worked for or with ends up overriding the SSL CA settings", would you at least enumerate some of your concerns here with examples?

@Lukasa

This comment has been minimized.

Show comment
Hide comment
@Lukasa

Lukasa Jan 28, 2016

Contributor

To be clear, with requests API in the form that it is in today pip cannot use the system trust store effectively, because it cannot pass both a CA directory and a CA file. It would need to, because it is not possible for pip to analytically determine whether a given path is a valid source of CAs without using OpenSSL extensively to try to load certs, which is a bizarre and onerous requirement.

Requests could in principle make that API change, but there is a question about whether we should do this properly more generally. As a team we are extremely reluctant to introduce different behaviours on different platforms, or to provide Linux users with a different logic than we provide Windows or OS X users.

However, despite your pretty negative opinion of us, we're not here to make your life difficult, and in fact we've been discussing this for a little while now at requests/requests#2966. However, for us to fix this in a way that is universally helpful and secure is not trivial.

Contributor

Lukasa commented Jan 28, 2016

To be clear, with requests API in the form that it is in today pip cannot use the system trust store effectively, because it cannot pass both a CA directory and a CA file. It would need to, because it is not possible for pip to analytically determine whether a given path is a valid source of CAs without using OpenSSL extensively to try to load certs, which is a bizarre and onerous requirement.

Requests could in principle make that API change, but there is a question about whether we should do this properly more generally. As a team we are extremely reluctant to introduce different behaviours on different platforms, or to provide Linux users with a different logic than we provide Windows or OS X users.

However, despite your pretty negative opinion of us, we're not here to make your life difficult, and in fact we've been discussing this for a little while now at requests/requests#2966. However, for us to fix this in a way that is universally helpful and secure is not trivial.

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Jan 28, 2016

Member

@Roguelazer maybe your concerns would be more productively be voiced on distros' bug trackers than here. If they can't get CA configuration right, then I struggle to reconcile why you trust them so steadfastly over anyone else.

Member

Ivoz commented Jan 28, 2016

@Roguelazer maybe your concerns would be more productively be voiced on distros' bug trackers than here. If they can't get CA configuration right, then I struggle to reconcile why you trust them so steadfastly over anyone else.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jan 28, 2016

Member

I don't believe we can reasonably use the system trust store in a cross platform way given the APIs that are available to us right now.

Some background: There are two "kinds" of trust paths that OpenSSL understands, there is CAPAth and there is CAFile, and OpenSSL gives two APIs for interacting with these one which gets a list of all of the paths and the other which instructs OpenSSL to load the default paths.

Previously we attempted to detect the system CA Bundle by simply enumerating a list of known file locations looking for one that existed. However that was causing issues in edge cases because there are a number of systems in the wild that have a stale (or possibly empty) ca file in one of those locations which happened to be the wrong location for that particular system (for example, the location for that OS was /foo/bar.pem but they also had a stale cert a /other/bar.pem and through luck that happened to be first in our list for detection). This lead people to having stale certificates which were not being updated being trusted instead of good ones.

So next we attempted to use the OpenSSL API to locate a CAFile, however it quickly was discovered that some distros have that API configured incorrectly and the CAFile points at the wrong location again (which again, may or may not exist and may or may not be stale) but they have a working CAPath.

Our next attempt was to switch to using just CAPath, however it was again discovered that some systems have an incorrectly defined CAPath that points to something that is empty and doesn't have any certificates in it.

Our next attempt was to switch to trying to utilize CAPath and if that didn't exist fall back to using CAFile. Again we discovered problems with this where some systems had a CAPath configured which existed, but which was not being managed as a OpenSSL CAPath and thus it had no actual cert files in it, which caused breakage again.

At this point I gave up and just went back to what we originally did, which was only trust our bundled copy unless certifi was installed, in which case it will use that instead.

A few other points that are important here:

  • Some systems, in order to properly use the system trust store require you to use both CAPath and CAFile and take a union of the certificates located in both. Currently requests does not allow us to path both, only one or the other.
  • Most Linux OSs are shipping the Mozilla CA bundle just as requests does, so the difference in default trust is often minimal if any, except for:
    • Certificates that the server operator has added or explicitly removed.
    • Unaudited CAs that some popular distros have in the past included (and might still be, haven't verified) (not a problem with requests bundle).
    • Incorrect usage of the Mozilla CA Bundle where they are trusting for HTTPS CA cerrtificates which are not valid for HTTPS but are for mail or something similar (not a problem with requests bundle).
  • OpenSSL does have another API, which simply instructs OpenSSL to load the default trust paths, this works assuming the OS has at least one location configured correctly. However we can't rely on it:
    • Currently requests does not make it possible to pass in your own SSLContext which is what we would need to call that API.
    • Not all OSs ship with CA certificates installed by default.
    • Not all OSs ship with CA certificates available at all.
    • When the OS isn't providing a CA bundle, we need to detect that, however while OpenSSL does let you inspect the loaded certificates if the source is a CAFile, if the source is a CAPath you can't see if there are any certificates available until you've begun to make requests. This means if we just blindly call the API to load the default locations, we may not (or we may, we don't know!) know if the system has any certificates at all.
  • Not all systems use OpenSSL at all so we need something that works for OS X and Windows as well.

All of this whole mess makes it very difficult to correctly utilize the trust store when we don't know ahead of time what OS we're going to be running on and, even assuming just systems that use OpenSSL, if they have a trust store available to them, installed by default, or in what ways their OpenSSL is going to be misconfigured. We attempted to do it, but it turns out that it takes a lot more effort to do it in a way that isn't going to add a lot of false errors (and false errors are very bad, they train users to ignore certificate validation errors).

Due to all of the above, I felt it was in the best interest of all of our users to return to the one method that we know will always work (because the file is bundled and is always there) and which we know the method used to update and generate the bundle are actually secure and accurate (unlike some OS vendors) and which does not include unaudited CAs by default (unlike some OS vendors). I feel this is an acceptable solution since the CA certificates are hardly the only security sensitive code here, and we're just as likely to need to patch a security sensitive bug as we are to need to do a quick update of the CA bundle and any of our users will want to handle both scenarios (made easier by the update nag that pip 6+ does).

If someone comes up with a good way to handle all of the edge cases from above I'm certainly open up to looking at a patch, but I'm wary of trying to do it because it's proven very difficult to do correctly thus far, and the error case tends to be "make a release, wait an hour and get people yelling at me that FooBar Linux can no longer use pip". I trust requests (and the team behind it) that, if they start enabling this to happen, that it will happen in a way that is both secure and user friendly and we'll of course pick that up automatically when we update requests and at that point we wouldn't disable it or anything.

Member

dstufft commented Jan 28, 2016

I don't believe we can reasonably use the system trust store in a cross platform way given the APIs that are available to us right now.

Some background: There are two "kinds" of trust paths that OpenSSL understands, there is CAPAth and there is CAFile, and OpenSSL gives two APIs for interacting with these one which gets a list of all of the paths and the other which instructs OpenSSL to load the default paths.

Previously we attempted to detect the system CA Bundle by simply enumerating a list of known file locations looking for one that existed. However that was causing issues in edge cases because there are a number of systems in the wild that have a stale (or possibly empty) ca file in one of those locations which happened to be the wrong location for that particular system (for example, the location for that OS was /foo/bar.pem but they also had a stale cert a /other/bar.pem and through luck that happened to be first in our list for detection). This lead people to having stale certificates which were not being updated being trusted instead of good ones.

So next we attempted to use the OpenSSL API to locate a CAFile, however it quickly was discovered that some distros have that API configured incorrectly and the CAFile points at the wrong location again (which again, may or may not exist and may or may not be stale) but they have a working CAPath.

Our next attempt was to switch to using just CAPath, however it was again discovered that some systems have an incorrectly defined CAPath that points to something that is empty and doesn't have any certificates in it.

Our next attempt was to switch to trying to utilize CAPath and if that didn't exist fall back to using CAFile. Again we discovered problems with this where some systems had a CAPath configured which existed, but which was not being managed as a OpenSSL CAPath and thus it had no actual cert files in it, which caused breakage again.

At this point I gave up and just went back to what we originally did, which was only trust our bundled copy unless certifi was installed, in which case it will use that instead.

A few other points that are important here:

  • Some systems, in order to properly use the system trust store require you to use both CAPath and CAFile and take a union of the certificates located in both. Currently requests does not allow us to path both, only one or the other.
  • Most Linux OSs are shipping the Mozilla CA bundle just as requests does, so the difference in default trust is often minimal if any, except for:
    • Certificates that the server operator has added or explicitly removed.
    • Unaudited CAs that some popular distros have in the past included (and might still be, haven't verified) (not a problem with requests bundle).
    • Incorrect usage of the Mozilla CA Bundle where they are trusting for HTTPS CA cerrtificates which are not valid for HTTPS but are for mail or something similar (not a problem with requests bundle).
  • OpenSSL does have another API, which simply instructs OpenSSL to load the default trust paths, this works assuming the OS has at least one location configured correctly. However we can't rely on it:
    • Currently requests does not make it possible to pass in your own SSLContext which is what we would need to call that API.
    • Not all OSs ship with CA certificates installed by default.
    • Not all OSs ship with CA certificates available at all.
    • When the OS isn't providing a CA bundle, we need to detect that, however while OpenSSL does let you inspect the loaded certificates if the source is a CAFile, if the source is a CAPath you can't see if there are any certificates available until you've begun to make requests. This means if we just blindly call the API to load the default locations, we may not (or we may, we don't know!) know if the system has any certificates at all.
  • Not all systems use OpenSSL at all so we need something that works for OS X and Windows as well.

All of this whole mess makes it very difficult to correctly utilize the trust store when we don't know ahead of time what OS we're going to be running on and, even assuming just systems that use OpenSSL, if they have a trust store available to them, installed by default, or in what ways their OpenSSL is going to be misconfigured. We attempted to do it, but it turns out that it takes a lot more effort to do it in a way that isn't going to add a lot of false errors (and false errors are very bad, they train users to ignore certificate validation errors).

Due to all of the above, I felt it was in the best interest of all of our users to return to the one method that we know will always work (because the file is bundled and is always there) and which we know the method used to update and generate the bundle are actually secure and accurate (unlike some OS vendors) and which does not include unaudited CAs by default (unlike some OS vendors). I feel this is an acceptable solution since the CA certificates are hardly the only security sensitive code here, and we're just as likely to need to patch a security sensitive bug as we are to need to do a quick update of the CA bundle and any of our users will want to handle both scenarios (made easier by the update nag that pip 6+ does).

If someone comes up with a good way to handle all of the edge cases from above I'm certainly open up to looking at a patch, but I'm wary of trying to do it because it's proven very difficult to do correctly thus far, and the error case tends to be "make a release, wait an hour and get people yelling at me that FooBar Linux can no longer use pip". I trust requests (and the team behind it) that, if they start enabling this to happen, that it will happen in a way that is both secure and user friendly and we'll of course pick that up automatically when we update requests and at that point we wouldn't disable it or anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment