SSL Certificate Bundles Aren't Found?

[Julian]

I know very little about the mechanisms here, so apologies if I've got this very wrong, but since portable PyPy vendors OpenSSL, and compiles it looking at /opt/prefix, it appears that at runtime no certificates are actually found from the system bundle, because that directory obviously will not exist on machines that use Portable PyPy.

I.e.,

>>> import ssl; ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/opt/prefix/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/opt/prefix/ssl/certs')

whereas the system certs are not there (this is CentOS 6, so they're in /etc/ssl/certs/ca-bundle.crt).

Setting SSL_CERT_FILE appears to be one way to fix the issue, but what's the actual recommendation for that?

[squeaky-pl]

One potential approach is to patch OpenSSL to use paths relative to the binary it is loaded from and bundle some ca-bundle.crt inside tarball. I can imagine it will make people concerned about security cringe. But I cannot rely on system-wide locations if i want to be portable.

[Julian]

Yeah -- you having to deal with it obviously isn't ideal, but not having access to the system bundle is fairly bad, no? I again know nothing here, but is there an option to just not statically link against OpenSSL? It seems possibly desirable for that on it's own too no? If there are OpenSSL vulnerabilities users will be in better shape to get the new patched version in place?

[posita]

This ain't my rodeo 🐮, but this seems like an OS-specific thing. For example on Debian and Ubuntu, certs are installed via the ca-certificates package (e.g., apt-get install ca-certificates). One probably doesn't want to be in the business of maintaining one's own bundled certs (see, e.g., this).

[posita]

As a work-around, perhaps rely on the user to set SSL_CERT_FILE environment variable to point the OS-installed ones?

[squeaky-pl]

OpenSSL is not statically linked in the tarball anymore, it is inside lib folder of the tarball as self-standing .so which is dynamically loaded. Sadly OpenSSL is one of the biggest offenders to portability and that's why it needs to be bundled. Each version regularly breaks versioning of symbols and ABI so all the dependant binaries needs to be recompiled. That's why many distributions choose to patch their shipped version of OpenSSL instead of upgrading to the latest one whenever there is new vulnerability discovered. I deliberately chose to bundle it because my main concern is out of the box experience. On the other side there are libraries like bzip2 which continue maintaing stable ABI for last 10 years and I dont need to bundle them.

[squeaky-pl]

To answer it from under angle - having portable binaries is very very unnatural on Linux, the whole ecosystem of distributions is tightly coupled around certain versions and hard-coded paths. But sometimes what you want is Windows-like experience and it's hard.

[squeaky-pl]

@Julian are you using "stock" system-wide certificate store or do you need it because you have some custom setup there?

[Julian]

Stock yeah.

Which is why to me, the current situation where stock system bundles aren't
going to work seems really nonideal :/
On Jun 16, 2015 02:36, "Squeaky" notifications@github.com wrote:

@Julian https://github.com/Julian are you using "stock" system-wide
certificate store or do you need it because you have some custom setup
there?

—
Reply to this email directly or view it on GitHub
#15 (comment)
.

[squeaky-pl]

I might try to apply a plan like this:

• research where the certificate store is located in commonly used distributions over last 10 years
• try to see if i can patch OpenSSL to make this actually work
• in case of not finding certificate store fallback to Mozilla certificates bundled inside the the portable distrubution

[pquerna]

This blog post provides an overview of the issue and the paths used on many common systems:

https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

As a work around, I've been doing this:

mkdir -p /opt/prefix/ssl
ln -nsf /etc/ssl/certs /opt/prefix/ssl/

And now portable-pypy will validate certificates correctly.

[squeaky-pl]

HI @Julian. After more than one year I finally decided to look into it.

I read PyPy's ssl module source and the CPython documentation on ssl module to be sure I understand well what's going on. I also got very useful information from the article that @pquerna linked. I came with a patch that tries looking at the locations that are mentioned in the article before falling back to bundled OpenSSL defaults (which obviously will never make sense). Finally if it doesnt find anything it will fall back to the latest ca bundle from certifi project.

The patch is quite simple:

--- lib-python/2.7/ssl.orig.py  2016-08-28 02:28:24.000000000 -0300
+++ lib-python/2.7/ssl.py   2016-09-06 14:33:42.917950566 -0300
@@ -278,6 +278,25 @@
             "subjectAltName fields were found")


+_cafile = None
+_capath = None
+
+# https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/
+def _find_cafile_and_capath():
+    global _cafile
+    global _capath
+
+    if not _cafile and not _capath:
+        if os.path.isfile('/etc/pki/tls/certs/ca-bundle.crt'):
+            _cafile = '/etc/pki/tls/certs/ca-bundle.crt'
+        elif os.path.isdir('/etc/ssl/certs'):
+            _capath = '/etc/ssl/certs'
+        else:
+            _cafile = os.path.dirname(__file__) + '/cacert.pem'
+
+    return _cafile, _capath
+
+
 DefaultVerifyPaths = namedtuple("DefaultVerifyPaths",
     "cafile capath openssl_cafile_env openssl_cafile openssl_capath_env "
     "openssl_capath")
@@ -285,14 +304,19 @@
 def get_default_verify_paths():
     """Return paths to default cafile and capath.
     """
-    parts = _ssl.get_default_verify_paths()
+    parts = list(_ssl.get_default_verify_paths())
+    cafile, capath = _find_cafile_and_capath()

     # environment vars shadow paths
-    cafile = os.environ.get(parts[0], parts[1])
-    capath = os.environ.get(parts[2], parts[3])
+    cafile = os.environ.get(parts[0], cafile)
+    capath = os.environ.get(parts[2], capath)
+
+    # overwrite what we get from bundled openssl since it's useless 
+    parts[1] = None
+    parts[3] = None

-    return DefaultVerifyPaths(cafile if os.path.isfile(cafile) else None,
-                              capath if os.path.isdir(capath) else None,
+    return DefaultVerifyPaths(cafile if os.path.isfile(cafile or '') else None,
+                              capath if os.path.isdir(capath or '') else None,
                               *parts)


@@ -389,7 +413,12 @@
         if sys.platform == "win32":
             for storename in self._windows_cert_stores:
                 self._load_windows_store_certs(storename, purpose)
-        self.set_default_verify_paths()
+       
+        if not os.environ.get('SSL_CERT_FILE') and not os.environ.get('SSL_CERT_DIR'):
+            locations = _find_cafile_and_capath()
+            self.load_verify_locations(*locations)  
+        else:
+            self.set_default_verify_paths()

It just tries two locations that happen to cover all the main distros (I tested it with Fedora, Centos, Debian, Ubuntu and OpenSUSE) and then fallbacks to bundled trust store from certifi. You can still supress this behavior by settings SSL_CERT_FILE and SSL_CERT_DIR. I think this gives us well-behaved out of the box experience without sacrifising security (it still looks at your distros default cert store).

I built a version of Portable PyPy with this patch so you can try it in your setup. This tarball also contains a small script to test certificte verifiction test_ssl_trust.py.

pypy-5.4-ssl-trust-linux_x86_64-portable/bin/pypy pypy-5.4-ssl-trust-linux_x86_64-portable/test_ssl_trust.py 
DefaultVerifyPaths(cafile='/etc/pki/tls/certs/ca-bundle.crt', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile=None, openssl_capath_env='SSL_CERT_DIR', openssl_capath=None)
OK https://www.python.org [Verified]
OK https://untrusted-root.badssl.com [Unverified]

Grab the patched version here: https://bitbucket.org/squeaky/portable-pypy/downloads/pypy-5.4-ssl-trust-linux_x86_64-portable.tar.bz2 and let me know what you think. Otherwise I am gonna merge this to next release version (which will be 5.4.1).

[squeaky-pl]

So, PyPy 5.4.1 happened faster then I thought, I am merging this to master

[Julian]

@squeaky-pl thanks a ton for getting to this -- I saw the message, but didn't read it carefully yet :)

[Julian]

@squeaky-pl given that I know very very little about this stuff clearly, I went to ask some people that do. (Thanks @dstufft @tiran @Lukasa).

I'm just quoting directly below, but the conclusion AIUI from Donald & Cory is "this list is probably too short, but regardless, other stuff has abandoned this approach because sometimes it turns out those files aren't certificate bundles".

 09:19:52        tos9| Hi.
 09:20:16        tos9| This is OT for pypi/cryptography I guess, apologies, but any chance any of you have a second to spot check
                       whether https://github.com/squeaky-pl/portable-pypy/issues/15#issuecomment-245059634 looks sane
 09:23:29  reaperhulk| hmm, not entirely off topic. Crys might want to take a look at that -- that's also a potential approach for
                       having functional trust roots while bundling our own openssl with manylinux1
 09:23:57  reaperhulk| (I have not reviewed it for correctness)
 09:24:14          * | tos9 nods
 09:24:29        tos9| I'm glad at least notionally that your reaction wasn't "uh no that's ridiculous" :)
 09:24:41  Crys| I don't think the list is long enough.
 09:24:55  Crys| some OS have a cafile, some a cafile
 09:25:27  lukasa| Yeah, if you're going to do that you should copy curl, which arguably has the longest list of search directories
 09:25:34  lukasa| But requests/urllib3 just gave up on the idea altogether.
 09:25:36          * | hawkowl deletes some cache dirs
 09:25:38  lukasa| dstufft has opinions here too
 09:25:46  Crys| sometimes the user wants to override them with env vars
 09:26:08  Crys| ssl.get_default_verify_paths() is your friend
 09:26:48        tos9| lukasa: Interesting. Any chance you have a link on why?
 09:27:23  Crys| the constants are compiled into OpenSSL and vary greatly between OS and distributions.
 09:28:05        tos9| Crys: (Clearly I am very out of my depth) -- but that's the point here isn't it?
 09:28:10  hawkowl| can someone just confirm that cryptography isn't broken on 3.3?
 09:28:10        tos9| He's vendoring openssl
 09:28:13  hawkowl| on linux
 09:28:16        tos9| And so what's there is wrong
 09:29:28      lukasa| tos9: Basically the short answer is that several distros have at one time or another shipped things that exist
                       at the path of a CA file or directory possibility that are not those things.
 09:29:39  lukasa| And OpenSSL doesn't have a function that you can point at a directory to say "load these or tell me that you can't"
 09:29:46  lukasa| So there's no way for us to know whether it worked or not.
 09:29:59  lukasa| And using all of them opens users up to an attack whereby we search somewhere they forgot to secure.
 09:30:05        tos9| A ha.
 09:30:09        tos9| OK that makes sense.
 09:30:11  lukasa| So we gave up on the whole thing and said "either explicitly tell us where your certs are, or we'll use certifi"
 09:30:51        Crys| tos9: the approach is also wrong because OpenSSL allows you to override the location with env vars.
 09:31:10        tos9| Crys: Besides SSL_CERT_FILE?
 09:31:26      lukasa| tos9: SSL_CERT_DIR is another
 09:31:28  lukasa| Woooooooo
 09:31:30        tos9| Right, both of those.
 09:31:43  Crys| and libressl removed the feature
 09:31:59        tos9| He claims both of those still work here
 09:32:15        tos9| And it looks like if I'm reading the patch correclty that they do to me
 09:32:17     dstufft| tos9: https://github.com/pypa/pip/pull/3416#issuecomment-176397615
         info: Stop trying to locate system trust stores by dstufft
 09:32:40        tos9| dstufft: Hooray! (There goes the mornin') -- thanks, having a read.
 09:33:56     dstufft| tos9: most of this has to do with trying to use the system OpenSSL, but part of that is pip has experience
                       with systems in the wild that have random certificate bundles in locations that would otherwise be a
                       "standard" location on another machine
 09:34:33  dstufft| and those random certificate bundles may or may not be getting updated at all (often times they are just a random
                    snapshot of whatever was available at some point in time) and may or not contain any certificates at all
 09:35:13        Crys| tos9: here is an old list: http://pastebin.com/bRNVwSt2
 09:35:39  Crys| on some OS you have to install extra packages
 09:36:17     dstufft| tos9: pip's gone through a few versions of handling SSL certificates, the first one was "attempt to locate a
                       CA bundle in a set of redefined paths, and if any of those exist use them, othewise fallback to bundled" and
                       that worked "OK" for some number of users, but for others it broke or left them insecure :/
 09:38:39  dstufft| (I would probably argue that curl's handling only works as well as it does, because the vast bulk of people are
                    not installing curl straight from curl, but via a redistributor who is making sure it's getting the right certs)
 09:39:54  lukasa| dstufft: It also skips the problem on two of the most awkward systems (OS X and Windows) by having support for
                   their TLS implementations out of the box.
 09:40:12  dstufft| lukasa: yea that too
 09:45:18        tos9| Yeah so "sometimes these paths aren't actually cert bundles" makes a lot of sense to me as a reason to not do
                       this
 09:45:58        tos9| dstufft, Crys, lukasa: do you mind if I quote this conversation on the ticket?
 09:46:09      lukasa| tos9: No objection from me.
 09:46:17     dstufft| tos9: fine with me
 09:46:28     dstufft| tos9: also sometimes they're cert bundles that nobody is updating!
 09:46:35  dstufft| it's basically impossible to tell :(
--------------------------------------------------------------------------------------------------------------------------------------
 09:48:21        Crys| tos9: I still consider certifi a bad approach. You want to use the system's cert store. An admin may have
                       installed additional certs (internal CA) or removed other certs for security reasons.
 09:49:06  lukasa| I consider certifi the best of a bad set of options currently, and until Crys writes me a ssl-API compatible set
                   of modules for SecureTransport and SChannel I'm not going to change it. ;)

[squeaky-pl]

Hi, thanks for the feedback. I looked at the conversation. This is the best effort I can do. It's still possible to override the paths with SSL_CERT_FILE and SSL_CERT_DIR with my patch. About searching the paths, if an attacker has an access to write to your file system you are already doomed because he can probably write to default OpenSSL location as well. Again, this is Windows-like exprience here. Otherwise you need to wait for your distro vendor provided pypy package. I will give it a shot and wait for somebody to point out if t didnt find the cert store. I will also add a paragraph about OpenSSL controversions to the README.

[Julian]

I think that sounds good to me, and obviously appreciated -- FWIW though, I don't think the point is that an attacker can write to those files. It's that in some places, those files just plain exist but are not certificate bundles, they're something else, and OpenSSL won't tell you that it failed at reading the file.

[squeaky-pl]

Hmm I tested it with Fedora, Centos, Debian, Ubuntu and OpenSUSE. Will probably test more distros and add them to supported "auto-discovery" distros in README paragraph about ssl. I will also look into curl behavior for more paths and mention that you should use SSL_CERT_FILE or SSL_CERT_DIR if you don't want this auto-discovery behavior.

[squeaky-pl]

I included a paragraph in the README about OpenSSL:

A word about OpenSSL
====================

This software bundles OpenSSL. Each build has a version of OpenSSL that was most recent and
 stable at the time of packaging this software. This is done because OpenSSL versions used across
 distrubtions in last 10 years greately vary and they are not compatible in ABI nor API way. This also 
means that if there is a major security issue with OpenSSL updating your system OpenSSL will not 
solve it for Portable PyPy. If you are looking for tight integration with your distribution you should 
probably wait until your distribution vendor packages version of PyPy you want to use or you can notify 
me and wait for a new build.

The `ssl` module will try to locate and use your system certificate store. 
Namely it will look for a `/etc/pki/tls/certs/ca-bundle.crt` file (RHEL derived systems) and then
 look for a `/etc/ssl/certs` directory (Debian dervied systems). Finally it will fallback to bundled
Mozilla trust stores extraced from `certifi` project. If you don't like this behavior or your
system trust store is located somewhere else you can use
`SSL_CERT_FILE` and `SSL_CERT_DIR` environment variables to point it somewhere else.

[squeaky-pl]

This has been ported to py3.5 branch. Nobody complained so I am closing this for now.