On ubuntu 16.04, "certificate verify failed" errors for www.google.com

[mohamedhafez]

I just did a apt dist-upgrade on my ubuntu 16.04 machine, and now when net::http tries to connect to www.google.com, it throws a "OpenSSL::SSL::SSLError: certificate verify failed". The ubuntu packages that got upgraded apparently affect the cert files and certificate verification, so i'm guessing that breaks jruby somehow:

ca-certificates 20170717~16.04.1
libidn11 1.32-3ubuntu1.2
libnss3 2:3.28.4-0ubuntu0.16.04.3
libnss3-nssdb 2:3.28.4-0ubuntu0.16.04.3

I thought it might be a bug with one of the packages, but curl still works without issue, and also the following code works in irb on ruby-2.3.0 on ubuntu, jruby 9.1.13.0 on mac, but not jruby 9.1.13.0 on my ubuntu machine:

require 'net/http'
uri = URI.parse("https://www.google.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.request(Net::HTTP::Get.new(uri.request_uri))

### Environment
JRuby version (jruby -v) and command line (flags, JRUBY_OPTS, etc):
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.131-b11 on 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11 +jit [linux-x86_64]

Operating system and platform (e.g. uname -a)
Linux momoserver 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

### Expected Behavior
www.google.com's ssl certificate should be verified without issue

### Actual Behavior
OpenSSL::SSL::SSLError: certificate verify failed

[mohamedhafez]

Also, when i provide my own cert_store to the Net::HTTP object, it also works without issue, so I'm guessing there must be something going wrong with JRuby's use of the default system cert store.

Also another point of clarification: while I'm getting the verify errors for google.com, I'm not getting the for other sites, like yahoo.com for example

[andreaseger]

we're seeing the same issue with both google and apple certificates

[nilsding]

The latest ca-certificates update threw out some Equifax and VeriSign certificates. jruby-openssl still tries to validate those as they are still present in the certificate chain of both apple.com and google.com (you can check the certificate chain for a site using the openssl tool: openssl s_client -showcerts -connect www.google.com:443 </dev/null).

In the meantime you could try to install the previous version of the ca-certificates package, I guess.

[mohamedhafez]

@nilsding so is this behavior incorrect of JRuby, since MRI and curl don't seem to do it?

[tmr08c]

Based on jruby/jruby-openssl#141 (comment), it looks like this may be an issue with jruby-openssl where it's unnecessarily checking further in the cert chain than it needs to.

Downgrading ca-certificates with sudo apt-get install ca-certificates=20160104ubuntu1 seems to work as a temporary solution.

Update In addition to downgrading, I also needed to update /etc/ca-certificates.conf by removing the ! in front of the Equifax listings. After doing this, I ran sudo update-ca-certificates to fetch the missing certificates.

[headius]

Usually these problems are just bad CA certs on the Java side compared to the native size (as MRI would use).

I assume this is latest jruby-openssl, yes?

jruby-openssl is supposed to also use system certs, but we've had challenges integrating the built-in "blessed" certs in JDK with ones from the system.

[mohamedhafez]

@headius yes I'm just using the default jruby-openssl that comes with JRuby 9.1.13.0, which looks like its the latest version, 0.9.21

[andreaseger]

Yes it happens with latest jruby-openssl.

And it correctly uses the system certs but in the system certs the equifax root cert was removed in the latest update of ca-certificates(or similar on other distributions).
Problem seems to be that the GeoTrust Global CA while in the trusted root certs itself is somehow also cross signed with that now invalid Equifax root cert. normal openssl seems to stop checking the chain after finding the trusted GeoTrust Gobal CA while jruby-openssl keeps checking the further certs in the chain.
(not quite deep enough into the whole topic to know how that cross signing actually works but assuming that openssl and for example windows is doing the correct thing for the google and apple certificates)

This issue is also reproducible on various different machines and distributions (given it already got the latest ca-certificates update).

[headius]

@andreaseger Hmmm so perhaps there's some config we can use for how deep to verify? From what you're saying, it sounds like JRuby's doing the right thing, but that ignores why native OpenSSL would do the "wrong" thing here, and if it's actually wrong.

[andreaseger]

Well I would not say that jruby-openssl is the only library which does the right thing. I would rather assume that the other libraries are doing the correct thing.
Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

This post https://www.mail-archive.com/openssl-dev@openssl.org/msg38674.html referenced in the jruby-openssl issue explains the situation quite well.

Seems to just be only now an issue here with the equifax certificate being revoked. And for apples certificates it's probably similar with one of the other certificates which got revoked in the recent updates.

Oh and regarding verification depth I have tried setting verify_depth( https://ruby-doc.org/stdlib-2.3.3/libdoc/net/http/rdoc/Net/HTTP.html#verify_depth ) but that seems to be ignored or is not working as intended. It would also only be a workaround in my opinion.

[mediocretes]

In the "are we crazy?" file, we also encountered this issue using phantom 2.1.1 on Ubuntu 16.04, and your instructions for downgrading temporarily fixed the problem for us.

I say temporarily because, hey, I want to pull those certs at some point, too.

We are not using jruby, I'm just commenting for solidarity's sake in response to:

Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

It's at least you and DocRaptor.

[headius]

@mediocrates Thanks for that! So it seems the cert change is causing problems for at least one other project.

[headius]

Well I'm downloading Ubuntu 16.04 to reproduce locally. I'll see if I can come up with anything.

[headius]

Well the good news is that 9.1.14 on fully-updated Ubuntu 16.04 was able to run @mohamedhafez's script without any errors. However the jruby-openssl issue is still open, so I'm not sure if this is fixed or my environment does not reflect yours.

Can someone post the version of ca-certificates that introduces the problem, so I can confirm I'm on the same version?

[nilsding]

@headius Does your /etc/ca-certificates.conf contain the following lines?

mozilla/Equifax_Secure_CA.crt
mozilla/Equifax_Secure_Global_eBusiness_CA.crt
mozilla/Equifax_Secure_eBusiness_CA_1.crt

If so, prefix those lines with a ! sign and run update-ca-certificates with a privileged user to reproduce this.

[mohamedhafez]

@headius it was ca-certificates 20170717~16.04.1

[headius]

Ok after updating ca-certificates to 20170717~16.04.1 I can indeed reproduce the issue.

This may not get fixed for 9.1.14 but we should be able to drop a jruby-openssl update shortly after.

[headius]

I can confirm that setting SSLContext.verify_depth to 0 does not solve the problem for me either.

I tried this via http.verify_depth, HTTP.start with keyword arg :verify_depth, and hardcoded in HTTP#connect. I have not confirmed the depth setting is actually getting into the right place in jruby-openssl.

[headius]

Ok more clues. When I pass -Djavax.net.debug=all to Java to dump out the whole process, I can see it does indeed try to verify Equifax because it sees that as the issuer for the GeoTrust certificate that's associated with Google's certificate.

When I check the root certificates on Ubuntu, all GeoTrust certs are issued by GeoTrust. However when I grep for Equifax in my JDK install (1.8u131, same as the original report), I do get hits.

I'm going to try to confirm that the GeoTrust cert in JDK 8u131 is still associated with Equifax. This may be a simple matter of out-of-date certs in OpenJDK.

[headius]

Yeah this is weird. I see the discrepancy on macos as well, but the Equifax certs have not been excluded there.

Here's keychain for the GeoTrust Global CA cert:

And here it is in the javax.net.debug log:

chain [2] = [
[
  Version: V3
  Subject: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 2762059360807314095743944092925343801268886471897734726827205372599492894886776968716511226505889655397481850507080643025642443194007248502440748624647559752206324612121434849632637734187975585119726040108049854460678876040724
332412792993061220100215761869148771363225170006518786596369272372091213539343886130277943218061361616722520651912317643036241026242970240486343490411672705520352450558095282433697964192353400557150441099729214476031795373906317835280968
0844232935574095508445145910310675421726257114605895831426222686272114090063230017292595425393719031924942422176213538487957041730136782988405751614792953
  public exponent: 65537
  Validity: [From: Mon May 20 23:00:00 CDT 2002,
               To: Mon Aug 20 23:00:00 CDT 2018]
  Issuer: OU=Equifax Secure Certificate Authority, O=Equifax, C=US
  SerialNumber: [    12bbe6]

I have been unable to figure out where it's getting this GeoTrust+Equifax cert.

[headius]

I have confirmed this still breaks under OpenJDK 8u151.

[headius]

I tried updating the Java CA cert for "GeoTrust Global CA" and it still seems to think it is issued by Equifax.

[headius]

Another attempt: I had forgotten that Debian-based dists try to share a cacerts across all JDKs, in the ca-certificates-java package and /etc/ssl/certs/java/cacerts. The latest package for 16.04 is a version from 2016, but I could not find any references to Experian in the cacerts file. I also tried to forcibly update the GeoTrust cert in that global cacerts, and it did not appear to help.

[headius]

Forcibly updating ca-certificates-java to the lastest Debian package (20170930) also does not appear to help.

[tomekw]

Maybe related to the issue I had 2 years ago? lostisland/faraday#371 (comment)

[headius]

@tomekw Hmm could be. I could see some sort of ordering problem causing a thing like this.

[headius]

I've continued exploration.

The logic in JRuby (jruby-openssl) is in StoreContext.verifyCertificate. This logic mimics what other SSL libraries do to acquire and verify the certificate chain.

The problem still seems to be that we're getting a bad certificate. Once the chain is prepared, the logic proceeds to confirm that the last cert in the chain is self-signed and in the certificate store, which would indicate it is in the trust root. In the current case, that certificate comes in as the GeoTrust/Equifax cert of unknown origin. Because it's not self-signed, we attempt to acquire its issuer's certificate, but the Equiface cert is no longer available.

So where is Equifax coming from?

Running strace against Java seems to show it only looking at the cacerts in my JDK install. When I dump that complete keystore, there's no references to Equifax.

[headius]

Ok, so I've confirmed that the certs coming from the Java keystore are correct: the root "GeoTrust Global CA" certificate is self-signed self-issued.

I've also figured out that Equifax is coming from the connection itself. Here's part of a packet read during handshake negotiation:

0800: 03 81 30 82 03 7D 30 82   02 E6 A0 03 02 01 02 02  ..0...0.........
0810: 03 12 BB E6 30 0D 06 09   2A 86 48 86 F7 0D 01 01  ....0...*.H.....
0820: 05 05 00 30 4E 31 0B 30   09 06 03 55 04 06 13 02  ...0N1.0...U....
0830: 55 53 31 10 30 0E 06 03   55 04 0A 13 07 45 71 75  US1.0...U....Equ
0840: 69 66 61 78 31 2D 30 2B   06 03 55 04 0B 13 24 45  ifax1-0+..U...$E
0850: 71 75 69 66 61 78 20 53   65 63 75 72 65 20 43 65  quifax Secure Ce
0860: 72 74 69 66 69 63 61 74   65 20 41 75 74 68 6F 72  rtificate Author
0870: 69 74 79 30 1E 17 0D 30   32 30 35 32 31 30 34 30  ity0...020521040
0880: 30 30 30 5A 17 0D 31 38   30 38 32 31 30 34 30 30  000Z..1808210400
0890: 30 30 5A 30 42 31 0B 30   09 06 03 55 04 06 13 02  00Z0B1.0...U....
08A0: 55 53 31 16 30 14 06 03   55 04 0A 13 0D 47 65 6F  US1.0...U....Geo
08B0: 54 72 75 73 74 20 49 6E   63 2E 31 1B 30 19 06 03  Trust Inc.1.0...
08C0: 55 04 03 13 12 47 65 6F   54 72 75 73 74 20 47 6C  U....GeoTrust Gl
08D0: 6F 62 61 6C 20 43 41 30   82 01 22 30 0D 06 09 2A  obal CA0.."0...*

So either we are supposed to do something else with this data that does not run up against the revoked Equifax cert, or we're making the request in such a way that Google is sending us bad cert info.

[headius]

I think I've finally figured out what we're doing wrong.

Google.com is sending a chain of certificates that still contains a "GeoTrust Global CA" certificate claiming that it was issued by Equifax. However the Equifax certificate is no longer trusted (in some contexts? all contexts?).

We are doing the right thing, for a pre-2015 OpenSSL clone. However, OpenSSL changed how they verify the certificate chain around 2015.

The change detailed above allows the verification logic to substitute a certificate from the trusted store for any untrusted cert coming back from the connection. In this case, we'd see that there's a better, newer cert for "GeoTrust Global CA" and use that one instead of the bad Equifax-issued cert Google is returning to us.

I'm not sure the best way to do this at the moment, but it would be part of the StoreContext.verifyCertificate method I mentioned earlier. I'm also still confused why Google (and apparently Apple and others) are still returning this bad root certificate.

[headius]

Well, this is progress. With the following patch, the test script works properly.

I seriously doubt I'm doing this right. My patch basically just eagerly searches through the incoming chain of certs, looks for matching DN in the trust store, and replaces the incoming cert with the trusted cert. That replaces the bad GeoTrust cert with a good one, and it works right.

The OpenSSL logic appears to do this lazily. I need to study it more to figure out the proper fix. If someone else wants to jump in, let me know :-)

diff --git a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
index 1e10880..3438edd 100644
--- a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
+++ b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
@@ -665,6 +665,19 @@ public class StoreContext {
         num = chain.size();
         x = chain.get(num - 1);
         depth = verifyParameter.depth;
+
+        List<X509Object> objects = store.getObjects();
+        for (int j = 0; j < sktmp.size(); j++) {
+            X509AuxCertificate cert = sktmp.get(j);
+            for (X509Object obj : objects) {
+                if (obj instanceof Certificate) {
+                    if (cert.cert.getSubjectDN().equals(((Certificate) obj).x509.getSubjectDN())) {
+                        sktmp.set(j, ((Certificate) obj).x509);
+                    }
+                }
+            }
+        }
+
         for(;;) {
             if ( depth < num ) break;

[headius]

Updated patch in jruby/jruby-openssl#149. This is a bit cleaner but needs some heavy review.

The logic in OpenSSL is spread all over, but it does appear to attempt to replace certs in the untrusted list with equivalent certs from the store before proceeding on to verification. My version does that for all certs in the untrusted list, in turn, until none are left or one gets replaced. Once replaces, the rest of the chain is tossed out, since we have a trusted root.

[headius]

This will get fixed in an update release to jruby-openssl, but not before the 9.1.14 release.

[headius]

This has not been updated by anyone in a long time...is there a chance that it has generally healed as old certs get dumped and CAs updated?

[mohamedhafez]

@headius Just retried with JRuby 9.1.17.0 on ubuntu 16.04 with all the latest packages, and everything is working properly again!

[nilsding]

yep, seems like both Apple and Google have updated their certificate chains.

[kares]

the certificate chain mess, seems to have been resolved. no reports at jossl's for a good while.