node v0.10.34 causes untrusted cert errors

[nicks]

Install the request module and run the following code:

var r = require('request')
r.get('https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-macosx.zip')

Expected behavior:
Fetches the file and exits

Actual behavior:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: CERT_UNTRUSTED
    at SecurePair.<anonymous> (tls.js:1381:32)
    at SecurePair.emit (events.js:92:17)
    at SecurePair.maybeInitFinished (tls.js:980:10)

This works as expected in Node v0.10.33, but fails in Node v0.10.34

[misterdjules]

Thank you very much for reporting this issue @nicks. This is due to the update of root certificates that was included in the 0.10.34 release.

Putting @indutny and @bnoordhuis in the loop since they're most familiar with this change.

[misterdjules]

@indutny If I add the /* Verisign Class 3 Public Primary Certification Authority */ root cert back into src/node_root_certs.h, I can't reproduce the issue.

[ploer]

Seems like this is fairly widespread; I'm seeing the same problem with amazon servers.

var r = require('request')
r.get('https://s3.amazonaws.com')

[indutny]

@misterdjules yeah, I just wanted to paste the same thing:

diff --git a/src/node_root_certs.h b/src/node_root_certs.h
index 336eeb5..3eddb68 100644
--- a/src/node_root_certs.h
+++ b/src/node_root_certs.h
@@ -88,6 +88,21 @@
 "2zsmWLIodz2uFHdh1voqZiegDfqnc1zqcPGUIWVEX/r87yloqaKHee9570+sB3c4\n"
 "-----END CERTIFICATE-----\n",

+/* Verisign Class 3 Public Primary Certification Authority */
+"-----BEGIN CERTIFICATE-----\n"
+"MIICPDCCAaUCEHC65B0Q2Sk0tjjKewPMur8wDQYJKoZIhvcNAQECBQAwXzELMAkGA1UEBhMC\n"
+"VVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAzIFB1YmxpYyBQ\n"
+"cmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2MDEyOTAwMDAwMFoXDTI4MDgw\n"
+"MTIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYD\n"
+"VQQLEy5DbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGf\n"
+"MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDJXFme8huKARS0EN8EQNvjV69qRUCPhAwL0TPZ\n"
+"2RHP7gJYHyX3KqhEBarsAx94f56TuZoAqiN91qyFomNFx3InzPRMxnVx0jnvT0Lwdd8KkMaO\n"
+"IG+YD/isI19wKTakyYbnsZogy1Olhec9vn2a/iRFM9x2Fe0PonFkTGUugWhFpwIDAQABMA0G\n"
+"CSqGSIb3DQEBAgUAA4GBALtMEivPLCYATxQT3ab7/AoRhIzzKBxnki98tsX63/Dolbwdj2ws\n"
+"qFHMc9ikwFPwTtYmwHYBV4GSXiHx0bH/59AhWM1pF+NEHJwZRDmJXNycAA9WjQKZ7aKQRUzk\n"
+"uxCkPfAyAw7xzvjoyVGM5mKf5p/AfbdynMk2OmufTqj/ZA1k\n"
+"-----END CERTIFICATE-----\n",
+
 /* Verisign Class 3 Public Primary Certification Authority - G2 */
 "-----BEGIN CERTIFICATE-----\n"
 "MIIDAjCCAmsCEH3Z/gfPqB63EHln+6eJNMYwDQYJKoZIhvcNAQEFBQAwgcExCzAJBgNVBAYT\n"

[indutny]

VeriSign Class 3 Public Primary CA
Description: Description: This root CA is the root used for Secure Site Pro Certificates ,
Premium SSL Certificates and Code Signing Certificates. It is intended to be the
primary root used for these products until Q4 2010 when VeriSign transitions to using
a 2048 bit root. After that transition this CA will be used as part of a cross certification
to ensure legacy applications continue to trust VeriSign certificates and must continue
to be included in root stores by vendors. This root is expected to be used in this way at
least until 12/31/2013 and vendors should not plan on removing support for this root
until officially advised that the root is no longer needed to support certificates or CRL
validation. 

[indutny]

@misterdjules it doesn't work in firefox anymore, but does work in chrome and safari. The test url is: https://ssltest23.bbtest.net/

[indutny]

Relevant mozilla page: https://blog.mozilla.org/security/2014/09/08/phasing-out-certificates-with-1024-bit-rsa-keys/

[indutny]

@misterdjules basically it has 1024-bit RSA private key so it is insecure. I guess we are doing the right job here and it doesn't need to be fixed.

[indutny]

@nicks @ploer this is not a final decision, but I'm afraid that this is the "right thing to do" about Amazon certificates. Technically, you could just use rejectUnauthorized: false to make it work. This won't provide any security risks since that 1024 bit certificate could be cracked in months, and probably already was (if someone ever needed it).

[nicks]

when you say "it doesn't work in firefox anymore", can you give more details? The link works fine for me on firefox 34, and Firefox claims the SSL certs are OK.

[nicks]

AFAICT, https://bbuseruploads.s3.amazonaws.com successfully accepts a 2048-bit cert on all browsers that I tried, but doesn't with node

[indutny]

@nicks You are right about bbuseruploads, checking it. Note that I was talking about 1024-bit CA cert, not the leaf bbuseruploads cert.

[shigeki]

It was pointed out at https://bugzilla.mozilla.org/show_bug.cgi?id=986005#c4 . For a temporary remedy, shigeki/node@604752c allows 1024-bit rootCA.

[shigeki]

@indutny Yes, it's working fine!

[dougwilson]

#8894 (comment) 👍

[indutny]

@shigeki @dougwilson here is the proper patch:

diff --git a/deps/openssl/openssl/crypto/x509/x509_vfy.c b/deps/openssl/openssl/crypto/x509/x509_vfy.c
index 920066a..c313024 100644
--- a/deps/openssl/openssl/crypto/x509/x509_vfy.c
+++ b/deps/openssl/openssl/crypto/x509/x509_vfy.c
@@ -238,6 +238,7 @@ int X509_verify_cert(X509_STORE_CTX *ctx)
     * is self signed.
     */

+retry:
    i=sk_X509_num(ctx->chain);
    x=sk_X509_value(ctx->chain,i-1);
    if (ctx->check_issued(ctx, x, x))
@@ -312,6 +313,11 @@ int X509_verify_cert(X509_STORE_CTX *ctx)
        {
        if ((chain_ss == NULL) || !ctx->check_issued(ctx, x, chain_ss))
            {
+           if (sk_X509_num(ctx->chain) > 1)
+               {
+               X509_free(sk_X509_pop(ctx->chain));
+               goto retry;
+               }
            if (ctx->last_untrusted >= num)
                ctx->error=X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY;
            else

[shigeki]

@indutny Yes. Are you going to submit this to upstream?

[indutny]

@shigeki working on it now

[shigeki]

Awesome

[dougwilson]

Nice. The patch to bring OpenSSL more in line with NSS is the better option than putting back in crappy root certs :) yay @indutny

[misterdjules]

Here's a quick summary of my understanding of some of the discussions we've been having so far in order to find the best way to solve this issue.

@indutny's change, while it seems reasonable and fixes this specific issue with s3.amazonaws.com, hasn't been tested thoroughly. Thus, releasing a new version with this change seems a bit too early.

For users who need to connect to s3.amazonaws.com with the 0.10.34 release, a workaround could be to specify the (now) missing unsafe CA certificate as an additional certificate to trust. The request module, and the built-in https and tls modules support that. Of course, there are critical security implications when doing that, and we would document them in details.

The current candidate fix could then be thoroughly tested and make it into the next stable release if and when we're confident that it doesn't break more sites than it fixes.

[shigeki]

I think it needs some time to deprecate 1024-bit rootCAs on node-v0.10. Reverting rootCA updates for now and releasing new node-v0.10.35 with announcement of forthcoming rootCA updates is better.

[hunterloftis]

Seeing tons of incidents at Heroku - we've locked latest stable to 0.10.33.

[albertpeiro]

Thanks @hunterloftis. Affected by this, fortunately it didn't reach production.

[albertpeiro]

I want to thank the collaborators for addressing this issue. And Heroku for
locking 0.10.33.

Looking at the future: how can we prevent this from happening again
unexpectedly? Could we build additional tests to ensure stability around
certificates?

It's a concern this made it to the public and cloud platforms deployed it
automatically. I see the difficulty of thinking in advance to this issue,
so what can we do now knowing about it?

Thank you.

On 21 December 2014 at 14:07, dmoore19 notifications@github.com wrote:

This issue is not limited to S3 but includes all api access to AWS
services(S3, Dynamo, SQS, etc). All my stuff stopped working.

—
Reply to this email directly or view it on GitHub
#8894 (comment).

Albert Peiró

[dmoore19]

Thanks Albert. I think a very simple AWS test would suffice. Send a file to S3 and throw a message on SQS would do the trick - 10 lines of test code. At least for AWS. I'm not sure about azure and others.

I would be willing to contribute an "AWS sanity check test" if you want.

Dave

On Dec 21, 2014, at 8:46 AM, Albert Peiró notifications@github.com wrote:

I want to thank the collaborators for addressing this issue. And Heroku for
locking 0.10.33.

Looking at the future: how can we prevent this from happening again
unexpectedly? Could we build additional tests to ensure stability around
certificates?

It's a concern this made it to the public and cloud platforms deployed it
automatically. I see the difficulty of thinking in advance to this issue,
so what can we do now knowing about it?

Thank you.

On 21 December 2014 at 14:07, dmoore19 notifications@github.com wrote:

This issue is not limited to S3 but includes all api access to AWS
services(S3, Dynamo, SQS, etc). All my stuff stopped working.

—
Reply to this email directly or view it on GitHub
#8894 (comment).

Albert Peiró
—
Reply to this email directly or view it on GitHub.

[chrisdickinson]

For the future, there's now a test that connects to AWS over https. If the cert is untrusted, the test will start failing.

[misterdjules]

Thank you all, fixed in 1425ccd.

[albertpeiro]

That's awesome. Thank you.

On 22 December 2014 at 19:54, Julien Gilli notifications@github.com wrote:

Closed #8894 #8894.

—
Reply to this email directly or view it on GitHub
#8894 (comment).

Albert Peiró

[kazuho]

FYI a new flag called X509_V_FLAG_TRUSTED_FIRST has been recently added to OpenSSL (presumably in 1.0.2 beta).

I have not tested it in node.js by myself, but the flag worked as a fix in a similar scenario.

[shigeki]

@kazuho Thanks. The patch of X509_V_FLAG_TRUSTED_FIRST can be easily backported to openssl 1.0.1. Though a new openssl-1.0.1k is to be released today, shigeki/node@joyent:v0.10...shigeki_20150108 is working fine to connect AWS S3. We cannot use 1024-bit root certificates forever. How about this? /CC @indutny @misterdjules

[indutny]

@bnoordhuis ^^

[ghost]

I have the same issue with node v0.10.31. I tried to update to v0.10.33 becuase that's next one is supposed to have the root certs change but it doesn't help either.

[VKhazinGladiator]

I am still getting the error.
The same certificate is accepted by browsers (chrome and firefox) on the same machine.

Ubuntu 14.04 TLS
node --version
v0.10.38

[Error: UNABLE_TO_VERIFY_LEAF_SIGNATURE]

Sorry the url is internal and cannot be accessed from the internet.

[misterdjules]

@VKhazinGladiator Without having access to some code that reproduces the issue and to the certificates used, it's going to be difficult to have an educated opinion about this problem.

Could you paste the detailed information about the whole certificate chain the server's certificate belongs to? Or could you reproduce a similar setup on a host that can be reached by anyone?

[VKhazinGladiator]

The issue has been traced to the certificate setup. How can I close the issue?

[misterdjules]

@VKhazinGladiator Thanks for the follow-up, the issue is already closed, not need to close it.

[adamjaffeback]

@VKhazinGladiator, we're pushing through this issue as well. Could you share what the certificate setup issue was?