Update SSL Certs #1659

Closed
kennethreitz opened this Issue Oct 8, 2013 · 64 comments

Comments

Projects
None yet
6 participants
Owner

kennethreitz commented Oct 8, 2013

We really need to automate this and have an official workflow and policy for security releases.

Fixes #1655

TODO:

  • Automate CA Bundle generation
  • Establish documented security policy

See Also:

Collaborator

sigmavirus24 commented Oct 9, 2013

👍

Collaborator

Lukasa commented Oct 11, 2013

Is the right thing to do here just to rewrite cURL's Perl script that converts the Mozilla text file into a .pem file? I took a look and it shouldn't be too awful to do.

Owner

kennethreitz commented Oct 11, 2013

the problem with mozilla's cert list is that it includes all of the blacklisted certs. We need to automate the removal of those.

The code at #1460 is interesting —it claims to be both a valid python file and a valid pem file at the same time.

Collaborator

Lukasa commented Oct 11, 2013

Oh, I didn't spot he had a parser for Mozilla's certdata. That's interesting. I'll take a look into using that as a springboard. /me assigns to self.

Lukasa was assigned Oct 11, 2013

Contributor

ncoghlan commented Oct 24, 2013

This came up during the review/discussion of PEP 453 (the now accepted PEP to add an ensurepip module to bootstrap pip in Python 3.4). However, that acceptance is currently conditional on meeting an integration timeline that includes resolving this issue as part of the PEP's security considerations: http://www.python.org/dev/peps/pep-0453/#security-considerations

While ensurepip isn't affected by this (since it doesn't talk to the internet), pip is affected when installing from PyPI (including to update itself).

The target date in the PEP is currently December 29th (a bit over two months from now)

Owner

kennethreitz commented Oct 24, 2013

/cc @dstufft

Owner

kennethreitz commented Oct 24, 2013

Will Pip be relying on Requests' CA Bundle (and to-be-introduced infrastructure)? I'm assuming so, since it needs to make calls out to the internet at large in addition to just PyPi.

Owner

kennethreitz commented Oct 24, 2013

What's the intended workflow for Pip? When a new cert gets blacklisted, will users need to run $ pip install pip --upgrade, or would an automatic CA Bundle update be preferred?

Contributor

dstufft commented Oct 24, 2013

Pip will be relying on the CA bundle shipped with requests at this point in time. I have a todo list item to see about the possibility of getting it to use system certificates (would that be something requests itself would be interested in?) but either way it'll need to use the bundled certs on AIX and HP-UX.

As of right now there is no plans for an automatic CA Bundle update, in order to get a new bundle they'll need to get a new pip.

Owner

kennethreitz commented Oct 24, 2013

@dstufft requests explicitly prefers not to rely on the system certs due to massive compatibility headaches the variance in distros often cause. I believe we have a utility function to enable this functionality built-in though. Will update shortly.

Owner

kennethreitz commented Oct 24, 2013

So, at this point in time, Pip is blocked only by our currently outdated CA Bundle, nothing else?

Contributor

dstufft commented Oct 24, 2013

On the requests side that's all we require for ensurepip yes. The development branch of pip already is using requests with the older ca bundle.

Owner

kennethreitz commented Oct 24, 2013

@dstufft one thing to clarify here: historically, we only provide security releases for the latest release. We now have rigid API stability, so this shouldn't effect Pip at all. But, it is something to be aware of.

Owner

kennethreitz commented Oct 24, 2013

e.g. if/when Requests v3.0.0 comes around (not currently roadmapped), I will recommend Pip upgrade to it.

Contributor

dstufft commented Oct 24, 2013

That sounds more or less reasonable as long as it doesn't remove anything we depend on :)

Owner

kennethreitz commented Oct 24, 2013

I'd rather not make this integration this rigid, but, if Pip wanted to provide an expected "api contract", it wouldn't be unwelcome.

Owner

kennethreitz commented Oct 24, 2013

The core of the SSL issue here is summed up in this conversation: http://twitter.theinfo.org/164062050971160576#id164067268924420098

Contributor

dstufft commented Oct 24, 2013

I doubt you'll remove any of it, off the top of my head it's timeouts, TLS verification, http proxy support, CONNECT proxy support, and some system for allowing us to treat file:// urls similarly to http urls (the adaper thing works fine for this).

Owner

kennethreitz commented Oct 24, 2013

@dstufft are you aware of any sort of alert system for cert blacklisting? Is it possible to automate this?

I may make a simple service for this.

Contributor

dstufft commented Oct 24, 2013

I'm not aware of anything.

FWIW, Christian Heimes mentioned to me that instead of blacklisting we want white listing based on the CKA_TRUST_* settings and enhanced key usage OIDs. Apparently there are certificates in the root store that are trusted, but not for HTTPS but only for code signing and such.

Contributor

dstufft commented Oct 24, 2013

agl is a security person on the Chromium project, here's something they wrote to extract mozilla's certificates https://github.com/agl/extract-nss-root-certs/blob/master/convert_mozilla_certdata.go

It's written in go though.

Contributor

ncoghlan commented Oct 24, 2013

Christian Heimes would be a good person to ping about this - he's been looking into a bunch of SSL issues for CPython, which is how the topic came up in relation to PEP 453 (see https://mail.python.org/pipermail/python-dev/2013-October/129755.html)

https://mail.python.org/pipermail/python-dev/2013-October/129738.html is the original post from Christian regarding cert management that prompted me to ask him to take a look at pip's SSL cert handling (which ended up resulting in the addition to the PEP 453 security considerations section already mentioned above)

http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.2.1.0.1.5.html has more details regarding what eGenix ended up doing for their cert bundles.

Owner

kennethreitz commented Oct 24, 2013

@dstufft interesting. What "transformations" need to be done against Mozilla's raw cert file?

http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt

I believe the workflow I used was taking the certs provided by curl's caextract and manually removing blacklisted certs.

Contributor

dstufft commented Oct 24, 2013

I'm trying to grok this go code to see what agl is doing to extract a list of trusted certificates.

Owner

kennethreitz commented Oct 24, 2013

Go is perfect if it functions properly. @dstufft if you're up for it, could you vet it?

Contributor

dstufft commented Oct 24, 2013

I'm reading it but I don't really know go, but to be honest agl is an actual expert in this area. I'd trust anything he had to say or wrote to do this way more then I'd trust myself.

Collaborator

sigmavirus24 commented Oct 24, 2013

You know who does know go, right? @lukasa that's who. ;)

Owner

kennethreitz commented Oct 24, 2013

@agl, any input you can provide would be much appreciated!

Owner

kennethreitz commented Oct 24, 2013

Going by the README, agl/extract-nss-root-certs appears to be perfect.

Contributor

dstufft commented Oct 24, 2013

Yea, it appears to be exactly what requests wants here. Wrap it up with a little something that just polls Mozilla and compares the list existing list of trusted against the polled list of trusted and you'll end up with a pretty nice solution. If there's anything wrong with extract-nss-root-certs it'd probably be reasonable to file bugs.

Owner

kennethreitz commented Oct 24, 2013

Tentative new cert is available here:

http://ci.kennethreitz.org/job/ca-bundle/lastSuccessfulBuild/artifact/certs.pem

+1 from @dstufft, @agl, or @tiran would be appreciated :)

Once I receive a +1, a new release will be cut.

Owner

kennethreitz commented Oct 24, 2013

So, the old CA Bundle is 211KB, and the new one is 1.4MB. This doesn't seem right :)

Owner

kennethreitz commented Oct 24, 2013

Oh, it looks like the go code didn't actually do anything.

Collaborator

Lukasa commented Oct 24, 2013

Really? Seems like on my machine it removed some untrusted certs. I'll build a gist to show you what I mean.

Contributor

tiran commented Oct 24, 2013

The code doesn't create a file with PEM encoded certificates...

Collaborator

Lukasa commented Oct 24, 2013

Apologies, because the files are massive, but they're here.

Collaborator

Lukasa commented Oct 24, 2013

An example, the explicitly distrusted certificate "MD5 Collisions Forged Rogue CA 25c3" is not present in the second file (or doesn't appear to be).

Collaborator

Lukasa commented Oct 24, 2013

@tiran Really? I don't know my PEM well enough, but it looks fine to me based on that Gist...

Contributor

tiran commented Oct 24, 2013

The file contains DER-encoded certificates in a octal notation. PEM certificates are base64 encoded DER blocks with limited line length and surrounded by

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

Collaborator

Lukasa commented Oct 24, 2013

I think my gist isn't very clear: the first file is the original, the second (at the bottom) is the output. Direct link here. That looks like PEM to me. =)

Contributor

tiran commented Oct 24, 2013

How exactly did you create the file? On my machine the go script doesn't return a PEM encoded file.

$ curl https://hg.mozilla.org/mozilla-central/raw-file/tip/security/nss/lib/ckfw/builtins/certdata.txt -o certdata.txt
$ go run convert_mozilla_certdata.go

Collaborator

Lukasa commented Oct 24, 2013

Oh, that's interesting. I used a different URL: https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1. This is the URL given in the README for that project.

I suspect the program is tripping up on a difference between the two files, but not logging the error.

Contributor

tiran commented Oct 24, 2013

That URL points to an outdated bundle from 2012

Collaborator

Lukasa commented Oct 24, 2013

Yeah, I just realised. =) Still, I think the diagnosis is solid. Just trying to work out what execution path we're taking through the Go code to make the mistake.

Collaborator

Lukasa commented Oct 24, 2013

Oh, this is obvious. To get the whole license block, the code looks through for the line beginning CVS_ID. That line no longer exists in the new certificate file. I'll see if I can tweak it.

Contributor

dstufft commented Oct 24, 2013

@tiran noticed the CVS_ID was missing, I guessed my way to https://gist.github.com/dstufft/7137007

Collaborator

Lukasa commented Oct 24, 2013

Collaborator

Lukasa commented Oct 24, 2013

Heh, probably don't need all three solutions. =D

Contributor

dstufft commented Oct 24, 2013

If three people arrive at the same conclusion, it must have a good chance of being right.. right?

Collaborator

Lukasa commented Oct 24, 2013

Agreed. =) I think that @tiran's solution is the best for Pull-Request action, because it's a lot smarter than mine. =) Gotta fix up the slightly inconsistent indentation in it though (L162).

Collaborator

Lukasa commented Oct 24, 2013

@kennethreitz If you're interested, I can try to wrap this code up into a nice binary that does all of the work itself: run one command and you're done. Sound good?

Owner

kennethreitz commented Oct 24, 2013

@lukasa i think this will work fine, but i won't stop you :)

Owner

kennethreitz commented Oct 24, 2013

My only concern now is being altered of new mistrusts.

Collaborator

Lukasa commented Oct 24, 2013

Seems like the easiest way to do that would be to have an automated job that runs this process overnight and diffs against the current version in the repository. Any trust changes = time for new security release.

Owner

kennethreitz commented Oct 24, 2013

I'll setup an alert system that just checks for changes in mozilla's file

Owner

kennethreitz commented Oct 24, 2013

We can now update this at any time by running $ make certs

kennethreitz referenced this issue Oct 24, 2013

Merged

v2.0.1 #1700

2 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment