New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On ubuntu 16.04, "certificate verify failed" errors for www.google.com #4802

Open
mohamedhafez opened this Issue Oct 3, 2017 · 35 comments

Comments

Projects
None yet
7 participants
@mohamedhafez
Contributor

mohamedhafez commented Oct 3, 2017

I just did a apt dist-upgrade on my ubuntu 16.04 machine, and now when net::http tries to connect to www.google.com, it throws a "OpenSSL::SSL::SSLError: certificate verify failed". The ubuntu packages that got upgraded apparently affect the cert files and certificate verification, so i'm guessing that breaks jruby somehow:

ca-certificates 20170717~16.04.1
libidn11 1.32-3ubuntu1.2
libnss3 2:3.28.4-0ubuntu0.16.04.3
libnss3-nssdb 2:3.28.4-0ubuntu0.16.04.3

I thought it might be a bug with one of the packages, but curl still works without issue, and also the following code works in irb on ruby-2.3.0 on ubuntu, jruby 9.1.13.0 on mac, but not jruby 9.1.13.0 on my ubuntu machine:

require 'net/http'
uri = URI.parse("https://www.google.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.request(Net::HTTP::Get.new(uri.request_uri))

### Environment
JRuby version (jruby -v) and command line (flags, JRUBY_OPTS, etc):
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.131-b11 on 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11 +jit [linux-x86_64]

Operating system and platform (e.g. uname -a)
Linux momoserver 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

### Expected Behavior
www.google.com's ssl certificate should be verified without issue

### Actual Behavior
OpenSSL::SSL::SSLError: certificate verify failed

@mohamedhafez

This comment has been minimized.

Show comment
Hide comment
@mohamedhafez

mohamedhafez Oct 3, 2017

Contributor

Also, when i provide my own cert_store to the Net::HTTP object, it also works without issue, so I'm guessing there must be something going wrong with JRuby's use of the default system cert store.

Also another point of clarification: while I'm getting the verify errors for google.com, I'm not getting the for other sites, like yahoo.com for example

Contributor

mohamedhafez commented Oct 3, 2017

Also, when i provide my own cert_store to the Net::HTTP object, it also works without issue, so I'm guessing there must be something going wrong with JRuby's use of the default system cert store.

Also another point of clarification: while I'm getting the verify errors for google.com, I'm not getting the for other sites, like yahoo.com for example

@andreaseger

This comment has been minimized.

Show comment
Hide comment
@andreaseger

andreaseger Oct 3, 2017

we're seeing the same issue with both google and apple certificates

andreaseger commented Oct 3, 2017

we're seeing the same issue with both google and apple certificates

@nilsding

This comment has been minimized.

Show comment
Hide comment
@nilsding

nilsding Oct 3, 2017

The latest ca-certificates update threw out some Equifax and VeriSign certificates. jruby-openssl still tries to validate those as they are still present in the certificate chain of both apple.com and google.com (you can check the certificate chain for a site using the openssl tool: openssl s_client -showcerts -connect www.google.com:443 </dev/null).

In the meantime you could try to install the previous version of the ca-certificates package, I guess.

nilsding commented Oct 3, 2017

The latest ca-certificates update threw out some Equifax and VeriSign certificates. jruby-openssl still tries to validate those as they are still present in the certificate chain of both apple.com and google.com (you can check the certificate chain for a site using the openssl tool: openssl s_client -showcerts -connect www.google.com:443 </dev/null).

In the meantime you could try to install the previous version of the ca-certificates package, I guess.

@mohamedhafez

This comment has been minimized.

Show comment
Hide comment
@mohamedhafez

mohamedhafez Oct 3, 2017

Contributor

@nilsding so is this behavior incorrect of JRuby, since MRI and curl don't seem to do it?

Contributor

mohamedhafez commented Oct 3, 2017

@nilsding so is this behavior incorrect of JRuby, since MRI and curl don't seem to do it?

@tmr08c

This comment has been minimized.

Show comment
Hide comment
@tmr08c

tmr08c Oct 3, 2017

Based on jruby/jruby-openssl#141 (comment), it looks like this may be an issue with jruby-openssl where it's unnecessarily checking further in the cert chain than it needs to.

Downgrading ca-certificates with sudo apt-get install ca-certificates=20160104ubuntu1 seems to work as a temporary solution.

Update In addition to downgrading, I also needed to update /etc/ca-certificates.conf by removing the ! in front of the Equifax listings. After doing this, I ran sudo update-ca-certificates to fetch the missing certificates.

tmr08c commented Oct 3, 2017

Based on jruby/jruby-openssl#141 (comment), it looks like this may be an issue with jruby-openssl where it's unnecessarily checking further in the cert chain than it needs to.

Downgrading ca-certificates with sudo apt-get install ca-certificates=20160104ubuntu1 seems to work as a temporary solution.

Update In addition to downgrading, I also needed to update /etc/ca-certificates.conf by removing the ! in front of the Equifax listings. After doing this, I ran sudo update-ca-certificates to fetch the missing certificates.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 4, 2017

Member

Usually these problems are just bad CA certs on the Java side compared to the native size (as MRI would use).

I assume this is latest jruby-openssl, yes?

jruby-openssl is supposed to also use system certs, but we've had challenges integrating the built-in "blessed" certs in JDK with ones from the system.

Member

headius commented Oct 4, 2017

Usually these problems are just bad CA certs on the Java side compared to the native size (as MRI would use).

I assume this is latest jruby-openssl, yes?

jruby-openssl is supposed to also use system certs, but we've had challenges integrating the built-in "blessed" certs in JDK with ones from the system.

@mohamedhafez

This comment has been minimized.

Show comment
Hide comment
@mohamedhafez

mohamedhafez Oct 4, 2017

Contributor

@headius yes I'm just using the default jruby-openssl that comes with JRuby 9.1.13.0, which looks like its the latest version, 0.9.21

Contributor

mohamedhafez commented Oct 4, 2017

@headius yes I'm just using the default jruby-openssl that comes with JRuby 9.1.13.0, which looks like its the latest version, 0.9.21

@andreaseger

This comment has been minimized.

Show comment
Hide comment
@andreaseger

andreaseger Oct 4, 2017

Yes it happens with latest jruby-openssl.

And it correctly uses the system certs but in the system certs the equifax root cert was removed in the latest update of ca-certificates(or similar on other distributions).
Problem seems to be that the GeoTrust Global CA while in the trusted root certs itself is somehow also cross signed with that now invalid Equifax root cert. normal openssl seems to stop checking the chain after finding the trusted GeoTrust Gobal CA while jruby-openssl keeps checking the further certs in the chain.
(not quite deep enough into the whole topic to know how that cross signing actually works but assuming that openssl and for example windows is doing the correct thing for the google and apple certificates)

This issue is also reproducible on various different machines and distributions (given it already got the latest ca-certificates update).

andreaseger commented Oct 4, 2017

Yes it happens with latest jruby-openssl.

And it correctly uses the system certs but in the system certs the equifax root cert was removed in the latest update of ca-certificates(or similar on other distributions).
Problem seems to be that the GeoTrust Global CA while in the trusted root certs itself is somehow also cross signed with that now invalid Equifax root cert. normal openssl seems to stop checking the chain after finding the trusted GeoTrust Gobal CA while jruby-openssl keeps checking the further certs in the chain.
(not quite deep enough into the whole topic to know how that cross signing actually works but assuming that openssl and for example windows is doing the correct thing for the google and apple certificates)

This issue is also reproducible on various different machines and distributions (given it already got the latest ca-certificates update).

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 4, 2017

Member

@andreaseger Hmmm so perhaps there's some config we can use for how deep to verify? From what you're saying, it sounds like JRuby's doing the right thing, but that ignores why native OpenSSL would do the "wrong" thing here, and if it's actually wrong.

Member

headius commented Oct 4, 2017

@andreaseger Hmmm so perhaps there's some config we can use for how deep to verify? From what you're saying, it sounds like JRuby's doing the right thing, but that ignores why native OpenSSL would do the "wrong" thing here, and if it's actually wrong.

@andreaseger

This comment has been minimized.

Show comment
Hide comment
@andreaseger

andreaseger Oct 4, 2017

Well I would not say that jruby-openssl is the only library which does the right thing. I would rather assume that the other libraries are doing the correct thing.
Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

This post https://www.mail-archive.com/openssl-dev@openssl.org/msg38674.html referenced in the jruby-openssl issue explains the situation quite well.

Seems to just be only now an issue here with the equifax certificate being revoked. And for apples certificates it's probably similar with one of the other certificates which got revoked in the recent updates.

Oh and regarding verification depth I have tried setting verify_depth( https://ruby-doc.org/stdlib-2.3.3/libdoc/net/http/rdoc/Net/HTTP.html#verify_depth ) but that seems to be ignored or is not working as intended. It would also only be a workaround in my opinion.

andreaseger commented Oct 4, 2017

Well I would not say that jruby-openssl is the only library which does the right thing. I would rather assume that the other libraries are doing the correct thing.
Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

This post https://www.mail-archive.com/openssl-dev@openssl.org/msg38674.html referenced in the jruby-openssl issue explains the situation quite well.

Seems to just be only now an issue here with the equifax certificate being revoked. And for apples certificates it's probably similar with one of the other certificates which got revoked in the recent updates.

Oh and regarding verification depth I have tried setting verify_depth( https://ruby-doc.org/stdlib-2.3.3/libdoc/net/http/rdoc/Net/HTTP.html#verify_depth ) but that seems to be ignored or is not working as intended. It would also only be a workaround in my opinion.

@mediocretes

This comment has been minimized.

Show comment
Hide comment
@mediocretes

mediocretes Oct 4, 2017

In the "are we crazy?" file, we also encountered this issue using phantom 2.1.1 on Ubuntu 16.04, and your instructions for downgrading temporarily fixed the problem for us.

I say temporarily because, hey, I want to pull those certs at some point, too.

We are not using jruby, I'm just commenting for solidarity's sake in response to:

Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

It's at least you and DocRaptor.

mediocretes commented Oct 4, 2017

In the "are we crazy?" file, we also encountered this issue using phantom 2.1.1 on Ubuntu 16.04, and your instructions for downgrading temporarily fixed the problem for us.

I say temporarily because, hey, I want to pull those certs at some point, too.

We are not using jruby, I'm just commenting for solidarity's sake in response to:

Again I'm not deep enough into the topic to make that decision but it seems like nobody else is suddenly claiming that the certificate of https://google.com or https://apple.com are now invalid.

It's at least you and DocRaptor.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

@mediocrates Thanks for that! So it seems the cert change is causing problems for at least one other project.

Member

headius commented Oct 30, 2017

@mediocrates Thanks for that! So it seems the cert change is causing problems for at least one other project.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Well I'm downloading Ubuntu 16.04 to reproduce locally. I'll see if I can come up with anything.

Member

headius commented Oct 30, 2017

Well I'm downloading Ubuntu 16.04 to reproduce locally. I'll see if I can come up with anything.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Well the good news is that 9.1.14 on fully-updated Ubuntu 16.04 was able to run @mohamedhafez's script without any errors. However the jruby-openssl issue is still open, so I'm not sure if this is fixed or my environment does not reflect yours.

Can someone post the version of ca-certificates that introduces the problem, so I can confirm I'm on the same version?

Member

headius commented Oct 30, 2017

Well the good news is that 9.1.14 on fully-updated Ubuntu 16.04 was able to run @mohamedhafez's script without any errors. However the jruby-openssl issue is still open, so I'm not sure if this is fixed or my environment does not reflect yours.

Can someone post the version of ca-certificates that introduces the problem, so I can confirm I'm on the same version?

@headius headius added this to the JRuby 9.1.14.0 milestone Oct 30, 2017

@nilsding

This comment has been minimized.

Show comment
Hide comment
@nilsding

nilsding Oct 30, 2017

@headius Does your /etc/ca-certificates.conf contain the following lines?

mozilla/Equifax_Secure_CA.crt
mozilla/Equifax_Secure_Global_eBusiness_CA.crt
mozilla/Equifax_Secure_eBusiness_CA_1.crt

If so, prefix those lines with a ! sign and run update-ca-certificates with a privileged user to reproduce this.

nilsding commented Oct 30, 2017

@headius Does your /etc/ca-certificates.conf contain the following lines?

mozilla/Equifax_Secure_CA.crt
mozilla/Equifax_Secure_Global_eBusiness_CA.crt
mozilla/Equifax_Secure_eBusiness_CA_1.crt

If so, prefix those lines with a ! sign and run update-ca-certificates with a privileged user to reproduce this.

@mohamedhafez

This comment has been minimized.

Show comment
Hide comment
@mohamedhafez

mohamedhafez Oct 30, 2017

Contributor

@headius it was ca-certificates 20170717~16.04.1

Contributor

mohamedhafez commented Oct 30, 2017

@headius it was ca-certificates 20170717~16.04.1

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Ok after updating ca-certificates to 20170717~16.04.1 I can indeed reproduce the issue.

This may not get fixed for 9.1.14 but we should be able to drop a jruby-openssl update shortly after.

Member

headius commented Oct 30, 2017

Ok after updating ca-certificates to 20170717~16.04.1 I can indeed reproduce the issue.

This may not get fixed for 9.1.14 but we should be able to drop a jruby-openssl update shortly after.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

I can confirm that setting SSLContext.verify_depth to 0 does not solve the problem for me either.

I tried this via http.verify_depth, HTTP.start with keyword arg :verify_depth, and hardcoded in HTTP#connect. I have not confirmed the depth setting is actually getting into the right place in jruby-openssl.

Member

headius commented Oct 30, 2017

I can confirm that setting SSLContext.verify_depth to 0 does not solve the problem for me either.

I tried this via http.verify_depth, HTTP.start with keyword arg :verify_depth, and hardcoded in HTTP#connect. I have not confirmed the depth setting is actually getting into the right place in jruby-openssl.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Ok more clues. When I pass -Djavax.net.debug=all to Java to dump out the whole process, I can see it does indeed try to verify Equifax because it sees that as the issuer for the GeoTrust certificate that's associated with Google's certificate.

When I check the root certificates on Ubuntu, all GeoTrust certs are issued by GeoTrust. However when I grep for Equifax in my JDK install (1.8u131, same as the original report), I do get hits.

I'm going to try to confirm that the GeoTrust cert in JDK 8u131 is still associated with Equifax. This may be a simple matter of out-of-date certs in OpenJDK.

Member

headius commented Oct 30, 2017

Ok more clues. When I pass -Djavax.net.debug=all to Java to dump out the whole process, I can see it does indeed try to verify Equifax because it sees that as the issuer for the GeoTrust certificate that's associated with Google's certificate.

When I check the root certificates on Ubuntu, all GeoTrust certs are issued by GeoTrust. However when I grep for Equifax in my JDK install (1.8u131, same as the original report), I do get hits.

I'm going to try to confirm that the GeoTrust cert in JDK 8u131 is still associated with Equifax. This may be a simple matter of out-of-date certs in OpenJDK.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Yeah this is weird. I see the discrepancy on macos as well, but the Equifax certs have not been excluded there.

Here's keychain for the GeoTrust Global CA cert:

image

And here it is in the javax.net.debug log:

chain [2] = [
[
  Version: V3
  Subject: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 2762059360807314095743944092925343801268886471897734726827205372599492894886776968716511226505889655397481850507080643025642443194007248502440748624647559752206324612121434849632637734187975585119726040108049854460678876040724
332412792993061220100215761869148771363225170006518786596369272372091213539343886130277943218061361616722520651912317643036241026242970240486343490411672705520352450558095282433697964192353400557150441099729214476031795373906317835280968
0844232935574095508445145910310675421726257114605895831426222686272114090063230017292595425393719031924942422176213538487957041730136782988405751614792953
  public exponent: 65537
  Validity: [From: Mon May 20 23:00:00 CDT 2002,
               To: Mon Aug 20 23:00:00 CDT 2018]
  Issuer: OU=Equifax Secure Certificate Authority, O=Equifax, C=US
  SerialNumber: [    12bbe6]

I have been unable to figure out where it's getting this GeoTrust+Equifax cert.

Member

headius commented Oct 30, 2017

Yeah this is weird. I see the discrepancy on macos as well, but the Equifax certs have not been excluded there.

Here's keychain for the GeoTrust Global CA cert:

image

And here it is in the javax.net.debug log:

chain [2] = [
[
  Version: V3
  Subject: CN=GeoTrust Global CA, O=GeoTrust Inc., C=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 2762059360807314095743944092925343801268886471897734726827205372599492894886776968716511226505889655397481850507080643025642443194007248502440748624647559752206324612121434849632637734187975585119726040108049854460678876040724
332412792993061220100215761869148771363225170006518786596369272372091213539343886130277943218061361616722520651912317643036241026242970240486343490411672705520352450558095282433697964192353400557150441099729214476031795373906317835280968
0844232935574095508445145910310675421726257114605895831426222686272114090063230017292595425393719031924942422176213538487957041730136782988405751614792953
  public exponent: 65537
  Validity: [From: Mon May 20 23:00:00 CDT 2002,
               To: Mon Aug 20 23:00:00 CDT 2018]
  Issuer: OU=Equifax Secure Certificate Authority, O=Equifax, C=US
  SerialNumber: [    12bbe6]

I have been unable to figure out where it's getting this GeoTrust+Equifax cert.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

I have confirmed this still breaks under OpenJDK 8u151.

Member

headius commented Oct 30, 2017

I have confirmed this still breaks under OpenJDK 8u151.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

I tried updating the Java CA cert for "GeoTrust Global CA" and it still seems to think it is issued by Equifax.

Member

headius commented Oct 30, 2017

I tried updating the Java CA cert for "GeoTrust Global CA" and it still seems to think it is issued by Equifax.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Another attempt: I had forgotten that Debian-based dists try to share a cacerts across all JDKs, in the ca-certificates-java package and /etc/ssl/certs/java/cacerts. The latest package for 16.04 is a version from 2016, but I could not find any references to Experian in the cacerts file. I also tried to forcibly update the GeoTrust cert in that global cacerts, and it did not appear to help.

Member

headius commented Oct 30, 2017

Another attempt: I had forgotten that Debian-based dists try to share a cacerts across all JDKs, in the ca-certificates-java package and /etc/ssl/certs/java/cacerts. The latest package for 16.04 is a version from 2016, but I could not find any references to Experian in the cacerts file. I also tried to forcibly update the GeoTrust cert in that global cacerts, and it did not appear to help.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 30, 2017

Member

Forcibly updating ca-certificates-java to the lastest Debian package (20170930) also does not appear to help.

Member

headius commented Oct 30, 2017

Forcibly updating ca-certificates-java to the lastest Debian package (20170930) also does not appear to help.

@tomekw

This comment has been minimized.

Show comment
Hide comment
@tomekw

tomekw Oct 30, 2017

Maybe related to the issue I had 2 years ago? lostisland/faraday#371 (comment)

tomekw commented Oct 30, 2017

Maybe related to the issue I had 2 years ago? lostisland/faraday#371 (comment)

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Oct 31, 2017

Member

@tomekw Hmm could be. I could see some sort of ordering problem causing a thing like this.

Member

headius commented Oct 31, 2017

@tomekw Hmm could be. I could see some sort of ordering problem causing a thing like this.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 2, 2017

Member

I've continued exploration.

The logic in JRuby (jruby-openssl) is in StoreContext.verifyCertificate. This logic mimics what other SSL libraries do to acquire and verify the certificate chain.

The problem still seems to be that we're getting a bad certificate. Once the chain is prepared, the logic proceeds to confirm that the last cert in the chain is self-signed and in the certificate store, which would indicate it is in the trust root. In the current case, that certificate comes in as the GeoTrust/Equifax cert of unknown origin. Because it's not self-signed, we attempt to acquire its issuer's certificate, but the Equiface cert is no longer available.

So where is Equifax coming from?

Running strace against Java seems to show it only looking at the cacerts in my JDK install. When I dump that complete keystore, there's no references to Equifax.

Member

headius commented Nov 2, 2017

I've continued exploration.

The logic in JRuby (jruby-openssl) is in StoreContext.verifyCertificate. This logic mimics what other SSL libraries do to acquire and verify the certificate chain.

The problem still seems to be that we're getting a bad certificate. Once the chain is prepared, the logic proceeds to confirm that the last cert in the chain is self-signed and in the certificate store, which would indicate it is in the trust root. In the current case, that certificate comes in as the GeoTrust/Equifax cert of unknown origin. Because it's not self-signed, we attempt to acquire its issuer's certificate, but the Equiface cert is no longer available.

So where is Equifax coming from?

Running strace against Java seems to show it only looking at the cacerts in my JDK install. When I dump that complete keystore, there's no references to Equifax.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 2, 2017

Member

Ok, so I've confirmed that the certs coming from the Java keystore are correct: the root "GeoTrust Global CA" certificate is self-signed self-issued.

I've also figured out that Equifax is coming from the connection itself. Here's part of a packet read during handshake negotiation:

0800: 03 81 30 82 03 7D 30 82   02 E6 A0 03 02 01 02 02  ..0...0.........
0810: 03 12 BB E6 30 0D 06 09   2A 86 48 86 F7 0D 01 01  ....0...*.H.....
0820: 05 05 00 30 4E 31 0B 30   09 06 03 55 04 06 13 02  ...0N1.0...U....
0830: 55 53 31 10 30 0E 06 03   55 04 0A 13 07 45 71 75  US1.0...U....Equ
0840: 69 66 61 78 31 2D 30 2B   06 03 55 04 0B 13 24 45  ifax1-0+..U...$E
0850: 71 75 69 66 61 78 20 53   65 63 75 72 65 20 43 65  quifax Secure Ce
0860: 72 74 69 66 69 63 61 74   65 20 41 75 74 68 6F 72  rtificate Author
0870: 69 74 79 30 1E 17 0D 30   32 30 35 32 31 30 34 30  ity0...020521040
0880: 30 30 30 5A 17 0D 31 38   30 38 32 31 30 34 30 30  000Z..1808210400
0890: 30 30 5A 30 42 31 0B 30   09 06 03 55 04 06 13 02  00Z0B1.0...U....
08A0: 55 53 31 16 30 14 06 03   55 04 0A 13 0D 47 65 6F  US1.0...U....Geo
08B0: 54 72 75 73 74 20 49 6E   63 2E 31 1B 30 19 06 03  Trust Inc.1.0...
08C0: 55 04 03 13 12 47 65 6F   54 72 75 73 74 20 47 6C  U....GeoTrust Gl
08D0: 6F 62 61 6C 20 43 41 30   82 01 22 30 0D 06 09 2A  obal CA0.."0...*

So either we are supposed to do something else with this data that does not run up against the revoked Equifax cert, or we're making the request in such a way that Google is sending us bad cert info.

Member

headius commented Nov 2, 2017

Ok, so I've confirmed that the certs coming from the Java keystore are correct: the root "GeoTrust Global CA" certificate is self-signed self-issued.

I've also figured out that Equifax is coming from the connection itself. Here's part of a packet read during handshake negotiation:

0800: 03 81 30 82 03 7D 30 82   02 E6 A0 03 02 01 02 02  ..0...0.........
0810: 03 12 BB E6 30 0D 06 09   2A 86 48 86 F7 0D 01 01  ....0...*.H.....
0820: 05 05 00 30 4E 31 0B 30   09 06 03 55 04 06 13 02  ...0N1.0...U....
0830: 55 53 31 10 30 0E 06 03   55 04 0A 13 07 45 71 75  US1.0...U....Equ
0840: 69 66 61 78 31 2D 30 2B   06 03 55 04 0B 13 24 45  ifax1-0+..U...$E
0850: 71 75 69 66 61 78 20 53   65 63 75 72 65 20 43 65  quifax Secure Ce
0860: 72 74 69 66 69 63 61 74   65 20 41 75 74 68 6F 72  rtificate Author
0870: 69 74 79 30 1E 17 0D 30   32 30 35 32 31 30 34 30  ity0...020521040
0880: 30 30 30 5A 17 0D 31 38   30 38 32 31 30 34 30 30  000Z..1808210400
0890: 30 30 5A 30 42 31 0B 30   09 06 03 55 04 06 13 02  00Z0B1.0...U....
08A0: 55 53 31 16 30 14 06 03   55 04 0A 13 0D 47 65 6F  US1.0...U....Geo
08B0: 54 72 75 73 74 20 49 6E   63 2E 31 1B 30 19 06 03  Trust Inc.1.0...
08C0: 55 04 03 13 12 47 65 6F   54 72 75 73 74 20 47 6C  U....GeoTrust Gl
08D0: 6F 62 61 6C 20 43 41 30   82 01 22 30 0D 06 09 2A  obal CA0.."0...*

So either we are supposed to do something else with this data that does not run up against the revoked Equifax cert, or we're making the request in such a way that Google is sending us bad cert info.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 2, 2017

Member

I think I've finally figured out what we're doing wrong.

Google.com is sending a chain of certificates that still contains a "GeoTrust Global CA" certificate claiming that it was issued by Equifax. However the Equifax certificate is no longer trusted (in some contexts? all contexts?).

We are doing the right thing, for a pre-2015 OpenSSL clone. However, OpenSSL changed how they verify the certificate chain around 2015.

The change detailed above allows the verification logic to substitute a certificate from the trusted store for any untrusted cert coming back from the connection. In this case, we'd see that there's a better, newer cert for "GeoTrust Global CA" and use that one instead of the bad Equifax-issued cert Google is returning to us.

I'm not sure the best way to do this at the moment, but it would be part of the StoreContext.verifyCertificate method I mentioned earlier. I'm also still confused why Google (and apparently Apple and others) are still returning this bad root certificate.

Member

headius commented Nov 2, 2017

I think I've finally figured out what we're doing wrong.

Google.com is sending a chain of certificates that still contains a "GeoTrust Global CA" certificate claiming that it was issued by Equifax. However the Equifax certificate is no longer trusted (in some contexts? all contexts?).

We are doing the right thing, for a pre-2015 OpenSSL clone. However, OpenSSL changed how they verify the certificate chain around 2015.

The change detailed above allows the verification logic to substitute a certificate from the trusted store for any untrusted cert coming back from the connection. In this case, we'd see that there's a better, newer cert for "GeoTrust Global CA" and use that one instead of the bad Equifax-issued cert Google is returning to us.

I'm not sure the best way to do this at the moment, but it would be part of the StoreContext.verifyCertificate method I mentioned earlier. I'm also still confused why Google (and apparently Apple and others) are still returning this bad root certificate.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 2, 2017

Member

Well, this is progress. With the following patch, the test script works properly.

I seriously doubt I'm doing this right. My patch basically just eagerly searches through the incoming chain of certs, looks for matching DN in the trust store, and replaces the incoming cert with the trusted cert. That replaces the bad GeoTrust cert with a good one, and it works right.

The OpenSSL logic appears to do this lazily. I need to study it more to figure out the proper fix. If someone else wants to jump in, let me know :-)

diff --git a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
index 1e10880..3438edd 100644
--- a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
+++ b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
@@ -665,6 +665,19 @@ public class StoreContext {
         num = chain.size();
         x = chain.get(num - 1);
         depth = verifyParameter.depth;
+
+        List<X509Object> objects = store.getObjects();
+        for (int j = 0; j < sktmp.size(); j++) {
+            X509AuxCertificate cert = sktmp.get(j);
+            for (X509Object obj : objects) {
+                if (obj instanceof Certificate) {
+                    if (cert.cert.getSubjectDN().equals(((Certificate) obj).x509.getSubjectDN())) {
+                        sktmp.set(j, ((Certificate) obj).x509);
+                    }
+                }
+            }
+        }
+
         for(;;) {
             if ( depth < num ) break;
Member

headius commented Nov 2, 2017

Well, this is progress. With the following patch, the test script works properly.

I seriously doubt I'm doing this right. My patch basically just eagerly searches through the incoming chain of certs, looks for matching DN in the trust store, and replaces the incoming cert with the trusted cert. That replaces the bad GeoTrust cert with a good one, and it works right.

The OpenSSL logic appears to do this lazily. I need to study it more to figure out the proper fix. If someone else wants to jump in, let me know :-)

diff --git a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
index 1e10880..3438edd 100644
--- a/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
+++ b/src/main/java/org/jruby/ext/openssl/x509store/StoreContext.java
@@ -665,6 +665,19 @@ public class StoreContext {
         num = chain.size();
         x = chain.get(num - 1);
         depth = verifyParameter.depth;
+
+        List<X509Object> objects = store.getObjects();
+        for (int j = 0; j < sktmp.size(); j++) {
+            X509AuxCertificate cert = sktmp.get(j);
+            for (X509Object obj : objects) {
+                if (obj instanceof Certificate) {
+                    if (cert.cert.getSubjectDN().equals(((Certificate) obj).x509.getSubjectDN())) {
+                        sktmp.set(j, ((Certificate) obj).x509);
+                    }
+                }
+            }
+        }
+
         for(;;) {
             if ( depth < num ) break;

headius added a commit to jruby/jruby-openssl that referenced this issue Nov 2, 2017

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 2, 2017

Member

Updated patch in jruby/jruby-openssl#149. This is a bit cleaner but needs some heavy review.

The logic in OpenSSL is spread all over, but it does appear to attempt to replace certs in the untrusted list with equivalent certs from the store before proceeding on to verification. My version does that for all certs in the untrusted list, in turn, until none are left or one gets replaced. Once replaces, the rest of the chain is tossed out, since we have a trusted root.

Member

headius commented Nov 2, 2017

Updated patch in jruby/jruby-openssl#149. This is a bit cleaner but needs some heavy review.

The logic in OpenSSL is spread all over, but it does appear to attempt to replace certs in the untrusted list with equivalent certs from the store before proceeding on to verification. My version does that for all certs in the untrusted list, in turn, until none are left or one gets replaced. Once replaces, the rest of the chain is tossed out, since we have a trusted root.

@headius headius modified the milestones: JRuby 9.1.14.0, JRuby 9.2.0.0 Nov 8, 2017

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Nov 8, 2017

Member

This will get fixed in an update release to jruby-openssl, but not before the 9.1.14 release.

Member

headius commented Nov 8, 2017

This will get fixed in an update release to jruby-openssl, but not before the 9.1.14 release.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius May 17, 2018

Member

This has not been updated by anyone in a long time...is there a chance that it has generally healed as old certs get dumped and CAs updated?

Member

headius commented May 17, 2018

This has not been updated by anyone in a long time...is there a chance that it has generally healed as old certs get dumped and CAs updated?

@headius headius modified the milestones: JRuby 9.2.0.0, JRuby 9.2.1.0 May 17, 2018

@mohamedhafez

This comment has been minimized.

Show comment
Hide comment
@mohamedhafez

mohamedhafez May 18, 2018

Contributor

@headius Just retried with JRuby 9.1.17.0 on ubuntu 16.04 with all the latest packages, and everything is working properly again!

Contributor

mohamedhafez commented May 18, 2018

@headius Just retried with JRuby 9.1.17.0 on ubuntu 16.04 with all the latest packages, and everything is working properly again!

@nilsding

This comment has been minimized.

Show comment
Hide comment
@nilsding

nilsding May 18, 2018

yep, seems like both Apple and Google have updated their certificate chains.

nilsding commented May 18, 2018

yep, seems like both Apple and Google have updated their certificate chains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment