making requests to https://google.com errors #4771

Closed
azylman opened this Issue Feb 14, 2013 · 41 comments
@azylman

This happens for me in node v0.8.12 and v0.8.19 on two different machines, one Ubuntu 12.04 and one OSX 10.8. It doesn't happen if I make requests to other domains.

var https = require('https');
https.request('https://google.com', function(res) {console.log(res)});

Produces:

events.js:68
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140463203215168:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:656:24)
    at SlabBuffer.use (tls.js:199:18)
    at CleartextStream.CryptoStream._push (tls.js:483:33)
    at SecurePair.cycle (tls.js:880:20)
    at SecurePair.cycle (tls.js:895:10)
    at EncryptedStream.CryptoStream.write (tls.js:267:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:93:17)
    at TCP.onread (net.js:396:14)
@bnoordhuis
Node.js Foundation member

Sorry, can't reproduce. Works for me with master and v0.8.

@bnoordhuis bnoordhuis closed this Feb 14, 2013
@rodrigoalvesvieira

@azylman Couldn't that be a problem with your local openssl program? Have you tested it for other HTTPS urls?

Tested here on Mac OSX Mountain Lion and everything is fine.

@azylman

@rodrigoalvesvieira I did test it with other HTTPS URLs and it only failed hitting Google. I'm not sure how it could be a problem with my local openssl program when I tried it on two different machines with two different versions of OpenSSL. Their versions of openssl were: OpenSSL 0.9.8r 8 Feb 2011 and OpenSSL 1.0.1 14 Mar 2012.

I asked one of my coworkers to confirm and he sees the same behavior, so this is not an isolated incident...

@bnoordhuis
Node.js Foundation member

Different machines but the same network? Check that there's not some kind of firewall or proxy causing interference.

@azylman

I've reproduced it on three different networks so far. On an AWS machine, on our office network, and also while tethering on Verizon's 4G network.

@AlexeyKupershtokh

I have the same issue when trying to work with google under xubuntu 12.10 and nodejs v0.8.19-0.8.20.

events.js:71
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140423179986752:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at SecurePair.cycle (tls.js:913:10)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)
@corywilkerson

Seeing this on Mountain Lion 10.8.2 and nodejs v0.8.20

> var options = { 
    hostname: 'gdata.youtube.com', 
    port: 443, 
    path:'/feeds/api/playlists/snippets?q=GoogleDevelopers', 
    headers: { 'User-Agent':'youtube-feeds.js (https://github.com/fvdm/nodejs-youtube)', 'Accept': 'application/json' }, 
    method: 'GET' 
}

> var request = https.request( options, function(res) { console.log(res) });

events.js:71
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140735290048896:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at SecurePair.cycle (tls.js:913:10)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)
@AlexeyKupershtokh

And Ubuntu 11.04 + node 0.8.19-1chl1~natty1 and 0.8.20-1chl1~natty1 (i386, from chris lea's ppa) on my office pc has the same issue in both cases from @corywilkerson and @azylman

Though Ubuntu 10.04.4 LTS + node v0.8.16-1chl1~lucid1 and 0.8.20-1chl1~lucid1 (amd64, also from chris lea's ppa) work well on a server located in a DC.

@AlexeyKupershtokh

Reopen maybe?

@bnoordhuis
Node.js Foundation member

For some reason it reproduces for me now (with google.com - but not www.google.com).

@bnoordhuis bnoordhuis reopened this Feb 19, 2013
@AlexeyKupershtokh

I've also noticed this. This could probably be related:

wicked@wicked-desktop:/$ wget -S https://google.com/
--2013-02-19 18:46:12--  https://google.com/
Resolving google.com... 74.125.143.138, 74.125.143.139, 74.125.143.100, ...
Connecting to google.com|74.125.143.138|:443... connected.
ERROR: certificate common name `*.google.com' doesn't match requested host name `google.com'.
To connect to google.com insecurely, use `--no-check-certificate'.

But this does not explain why does node still crash on an https://play.google.com/ url.

@abarron87

I am getting the same issue with the YouTube data API, it worked yesterday afternoon but coming in the morning I see:

Error: 140735247131008:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288

Running node 0.8.15

Hostname: "gdata.youtube.com"
Path: "/feeds/api/videos?q=arsenal&orderby=relevance&v=2&alt=json&max-results=2&key="+apiKey

Requests work in the browser.

@agl

Reproducibility for this issue will vary because of differing configs across our frontend clusters and for different domains. However, over time, I expect increasing amounts of Google and non-Google services to stop working because of this.

The wget issue noted above is due to a deficiency in wget (lack of SNI support) and is unrelated.

I think the problem arises from a failure to clear the OpenSSL error stack in node_crypto.cc:

EVP_PKEY *pkey = NULL;
RSA *rsa = NULL;
if( NULL != (pkey = X509_get_pubkey(peer_cert))
    && NULL != (rsa = EVP_PKEY_get1_RSA(pkey)) ) {
    BN_print(bio, rsa->n);

When EVP_PKEY_get1_RSA fails it will leave an error on the stack and that appears to be popping up later in processing.

I cannot reproduce this on master and I bisected the fix to d59beb9. However, that's a wider ranging change that likely fixed the issue by accident or, possibly, masked it for a little longer.

As a smaller change that can be back-ported, I'd suggest checking that the public key is an RSA key before calling EVP_PKEY_get1_RSA:

EVP_PKEY *pkey = NULL;
RSA *rsa = NULL;
if( NULL != (pkey = X509_get_pubkey(peer_cert))
    && EVP_PKEY_RSA == EVP_PKEY_id(pkey)
    && NULL != (rsa = EVP_PKEY_get1_RSA(pkey)) ) {
    BN_print(bio, rsa->n);

(Note the extra clause in the condition.)

That fixes the proximal issue for me although the problem of not cleaning the OpenSSL error stack appears to be more pervasive.

@lukebayes lukebayes pushed a commit that referenced this issue Feb 21, 2013
Luke Bayes Fixed unexpected non-RSA verification failures.
This fix ensures that the RSA verification is only used for RSA signatures.

nodejs/node-v0.x-archive#4771
e6ce416
@alexanderturner

Still having this issue on v0.8.21, currently pushing to google over HTTP, though HTTPS is still failing with:
Error: 140692638627648:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

@bnoordhuis bnoordhuis was assigned Feb 27, 2013
@isaacs

@bnoordhuis Any idea when you'll get a chance to look into this? I think it's causing some occasional errors talking to the npm registry as well.

@bnoordhuis
Node.js Foundation member

I have a preliminary patch but it seems to slightly regress make bench-tls. That's on a MBA though so take with some salt.

diff --git a/src/node_crypto.cc b/src/node_crypto.cc
index 187b344..44d2171 100644
--- a/src/node_crypto.cc
+++ b/src/node_crypto.cc
@@ -901,6 +901,16 @@ int Connection::HandleBIOError(BIO *bio, const char* func, int rv) {


 int Connection::HandleSSLError(const char* func, int rv, ZeroStatus zs) {
+  // Forcibly clear OpenSSL's error stack on return. This stops stale errors
+  // from popping up later in the lifecycle of the SSL connection where they
+  // would cause spurious failures. It's a rather blunt method, though.
+  // ERR_clear_error() isn't necessarily cheap either.
+  struct ClearErrorOnReturn {
+    ~ClearErrorOnReturn() { ERR_clear_error(); }
+  };
+  ClearErrorOnReturn clear_error_on_return;
+  (void) &clear_error_on_return;  // Silence unused variable warning.
+
   if (rv > 0) return rv;
   if ((rv == 0) && (zs == kZeroIsNotAnError)) return rv;
@bnoordhuis bnoordhuis added a commit that closed this issue Feb 27, 2013
@bnoordhuis bnoordhuis crypto: clear error stack
Clear OpenSSL's error stack on return from Connection::HandleSSLError().
This stops stale errors from popping up later in the lifecycle of the
SSL connection where they would cause spurious failures.

This commit causes a 1-2% performance regression on `make bench-tls`.
We'll address that in follow-up commits if possible but let's ensure
correctness first.

Fixes #4771.
c6e2db2
@jfromaniello

is there a 0.8.x not affected by this issue? I don't know what happen, maybe google changed something but I started to see this error randomly few days ago and I can't find a version not affected.

I am not sure how to workaround it

If not, any ETA of a new version with the fix?

thanks a bunch

@bnoordhuis bnoordhuis added a commit to bnoordhuis/node that referenced this issue Mar 4, 2013
@bnoordhuis bnoordhuis crypto: clear error stack
Clear OpenSSL's error stack on return from Connection::HandleSSLError().
This stops stale errors from popping up later in the lifecycle of the
SSL connection where they would cause spurious failures.

This commit causes a 1-2% performance regression on `make bench-tls`.
We'll address that in follow-up commits if possible but let's ensure
correctness first.

Fixes #4771.

This is a back-port of commit c6e2db2 from the master branch.

Conflicts:
	src/node_crypto.cc
1e7b3d5
@bnoordhuis
Node.js Foundation member

The patch hasn't been back-ported to v0.8 yet, it needs a little more testing for unforeseen side effects. The back-port commit is bnoordhuis/node@1e7b3d5 in case you want to try it out.

@jfromaniello

Thanks, I will try it. I have been testing my use case with all 0.8.x releases and 0.8.1, 0.8.2, and 0.8.3 seems to not have this problem. Version 0.8.4 to 0.8.21 (included) fails.

The url I am using in my application is https://apps-apis.google.com/a/feeds...

@bnoordhuis
Node.js Foundation member

Hm, that would suggest that eb2ca10 is the culprit. Can you connect when you set { rejectUnauthorized: false } in the options object?

@jfromaniello

I tried with { rejectUnauthorized: false } and still same issue:

[Error: 1077155136:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288

I went into tls.js, the validation and the regex (changed in that commit you point me) seems to work ok..

@bnoordhuis
Node.js Foundation member

Okay, in that case the back-ported patch should fix it for you.

@angelochen960

in OS X mountain lion, Node 0.8.18 and 0.8.21, a query to youtube api, sometimes it works, sometimes, got following error:

https://gdata.youtube.com/feeds/api/videos/wzzUjm8apzA?v=2&alt=json
[Error: 140735083155840:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

any idea how to fix this? Thanks,

@bnoordhuis
Node.js Foundation member

Have you tried the patch? (Hint: "no" is not an acceptable answer. "Yes, but it doesn't work" however is.)

@angelochen960

sorry, I did not, how to apply the patch? in OS X, I just run the standard installer from dmg.

@bnoordhuis
Node.js Foundation member

Try this:

$ git clone git://github.com/joyent/node.git && cd node
$ git checkout origin/v0.8
$ curl -s https://github.com/bnoordhuis/node/commit/1e7b3d5.patch | git am
$ ./configure && make

The binary ends up in out/Release, you can install it with sudo make install.

@bnoordhuis
Node.js Foundation member

Oh, and add -j<cores> if you want to speed up the build. Replace <cores> with the number of CPUs in your machine.

@angelochen960

thanks, tried the patch, it works, then again I went back to 0.8.21, it works as well, I have to try some more to see if 0.8.21 will have that error again, then I will re-try the patch, looks like an intermittent error.

@benbuckman

After installing 0.8.22 from source with the 1e7b3d5 cherry-picked, I'm still getting errors:

Error: 140735117736288:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:683:

at CleartextStream._puller (tls.js:667:24)
at CleartextStream.CryptoStream._pull (tls.js:601:19)
at SecurePair.cycle (tls.js:891:20)
at EncryptedStream.CryptoStream.write (tls.js:285:13)
at Socket.ondata (stream.js:38:26)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:397:14)

when I request('https://google.com'), request('https://yahoo.com'), or my personal site on https.

(On OSX 10.7.5)

@bnoordhuis
Node.js Foundation member

@newleafdigital If that's the request method from the request module from npm, please open a bug report there. If it's the request method from the built-in http module, please post a test case because I can't reproduce it.

@michalmanko

Hey, I have the same problem, simple test case:

var https = require('https');

var url = 'https://www.youtube.com/watch?v=2I0P_Dpd5xo'; // doesn't work
// url = 'https://google.com'; // doesn't work
// url = 'https://www.facebook.com'; // it works
// url = 'https://github.com'; // it works
// url = 'https://login.yahoo.com/'; // it works
var href = require('url').parse(url);

var request = https.request(href, function(res) {
        console.log("statusCode: ", res.statusCode);
        console.log("headers: ", res.headers);
        res.on('data', function(d) {
                process.stdout.write(d);
        });
});
request.end();
request.on('error', function(e) {
        throw e;
});

and the exception:

nodeHttpsError.js:19
        throw e;
              ^
Error: 139698244785984:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)

I tested it on several environments:

  • Ubuntu 12.04.2 + Node 0.8.22 + OpenSSL 1.0.1 - Network A - doesn't work
  • Ubuntu 12.04.2 + Node 0.8.21 + OpenSSL 1.0.1 - Network A - doesn't work
  • Ubuntu 10.04.4 + Node 0.8.20 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 10.04.4 + Node 0.8.18 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 10.04.4 + Node 0.8.22 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 12.04.2 + Node 0.8.16 + OpenSSL 1.0.1 - Network B - works

It looks strange, in my case it could be one of two problems:

  • A bug is in new Node versions (>0.8.16) and only on Ubuntu 12
  • Network configuration is the problem (?)

I will try to upgrade last machine to the newest Node version and check it out, but not today.

If you need more info, feel free to ask.

Update:
One more server:

  • Ubuntu 12.04.2 + Node 0.8.22 + OpenSSL 1.0.1 - Network B - works
@bnoordhuis
Node.js Foundation member

I can confirm that the issue exists with v0.8 but not v0.10 and that the proposed patch or upgrading openssl doesn't fix it.

Interestingly enough, git bisect pinpoints c6e2db2 as the commit that fixes it.

@bnoordhuis
Node.js Foundation member

upgrading openssl doesn't fix it

However, applying the patch and linking to the 0.98 system openssl does fix it. Hm.

@bnoordhuis
Node.js Foundation member

It seems the bug is in (or triggered by) lib/http.js or lib/https.js because a plain TLS connection works. That is, the following test case works:

var tls = require('tls');
var conn = tls.connect(443, 'www.youtube.com', function() {
  conn.pipe(process.stdout);
  conn.write('GET /watch?v=2I0P_Dpd5xo HTTP/1.1\r\n' +
             'Host: www.youtube.com\r\n' +
             '\r\n');
});
@bnoordhuis bnoordhuis reopened this Mar 13, 2013
@bnoordhuis bnoordhuis added a commit that referenced this issue Mar 13, 2013
@bnoordhuis bnoordhuis crypto: check key type in GetPeerCertificate()
Works around the following exception:

  Error: 140463203215168:error:0607907F:digital envelope
  routines:EVP_PKEY_get1_RSA:expecting an rsa key:
  ../deps/openssl/openssl/crypto/evp/p_lib.c:288:
    at CleartextStream._pusher (tls.js:656:24)
    at SlabBuffer.use (tls.js:199:18)
    at CleartextStream.CryptoStream._push (tls.js:483:33)
    at SecurePair.cycle (tls.js:880:20)
    <snip>

The issue has been solved properly in v0.10 and the master branch as of
commit c6e2db2 ("crypto: clear error stack"). This is the "back-port"
to v0.8.

For some (rather unquantifiable) reason the original fix only works for
the tls module in v0.8 but not the https module unless OpenSSL is
downgraded to 0.9.8. Upgrading OpenSSL does *not* fix it, however.

The https module doesn't appear to be at fault; upgrading it to v0.10
doesn't fix the issue. That leaves either the tls or the http module
(that https derives from) but the changes to those modules are too
massive to back-port as-is.

`git bisect` over the v0.8 -> v0.10 commits didn't show up anything
useful, it pinpoints c6e2db2 as the commit that fixes things.

I've spent several hours on this now and seeing that v0.8 is in
maintenance mode, this cheap hack will have to do.

Fixes #4771.
2c41a80
@bnoordhuis
Node.js Foundation member

'Fixed' in v0.8 per 2c41a80. I've tried a lot of things but ended up settling for @agl's hack...

@bnoordhuis bnoordhuis closed this Mar 13, 2013
@fvdm fvdm referenced this issue in fvdm/nodejs-youtube Mar 16, 2013
@fvdm fvdm Replaced HTTPS with HTTP to fix `connection error`
This module does not handle user credentials.

> Error: 140735110496640:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:
1e34663
@fvdm fvdm referenced this issue in fvdm/nodejs-youtube Mar 16, 2013
Closed

Update npm package to reflect recent bug fixes #3

@fvdm fvdm added a commit to fvdm/nodejs-youtube that referenced this issue Mar 16, 2013
@fvdm fvdm Avoiding Node 0.8.4-0.8.22 due to SSL bug (#3)
Node versions 0.8.4-0.8.22 appear to be affected by this bug, therefore the module is now dependent on earlier and later versions.

Refer to ticket nodejs/node-v0.x-archive#4771 for details.
9a4dad1
@fightingtheboss fightingtheboss referenced this issue in meteor/meteor Mar 21, 2013
Closed

SSL bug in Node 0.8.4 - 0.8.22 #859

@thedufer

@agl Can you explain/link to the changes google made that caused this? Presumably if it mattered which frontend we hit, this bug was triggered by something on your end.

@jparkrr jparkrr referenced this issue in Singly/hallway Mar 29, 2013
Merged

Temp gdata http #861

@bowdev bowdev pushed a commit to bowdev/arrow that referenced this issue Dec 3, 2013
aresyhoo Provide default port 443 when do https request
Due to a bug in nodejs Node 0.8.4 - 0.8.22, according to
nodejs/node-v0.x-archive#4771

We need add port 443 as default in our own https tests.
7900b30
@bowdev bowdev pushed a commit to bowdev/arrow that referenced this issue Dec 3, 2013
aresyhoo Support https port 443
To workaround a SSL bug in https module for node 0.8.4-0.8.22 as
mentioned at nodejs/node-v0.x-archive#4771
475e5d6
@bowdev bowdev referenced this issue in yahoo/arrow Dec 3, 2013
Merged

Support https port 443 #152

@grantjbutler grantjbutler added a commit to grantjbutler/spikebot that referenced this issue Jan 9, 2014
@grantjbutler grantjbutler Fixed some potential HTTPS request issues outlined here: nodejs/node-…
…v0.x-archive#4771 . Fix one last formatting thing for time. Dang it, JavaScript
72370a1
@5punk 5punk referenced this issue in Atraci/Atraci Aug 3, 2014
Closed

Atraci stops playing on "auto" song change #42

@elierotenberg

I'm having the same problem on node v0.11.9 x86 with --harmony, Windows 7.
https request work for some domains (eg. github.com) but not for others (eg. graph.facebook.com).

@ghost

The changes that caused this were the introduction of ECDSA keys. I'm still at a loss as to why now the latest stable version works when really the proper fix wasn't applied. The issue stems from using the EVP_PKEY_get1_RSA function against a possible ECDSA certificate. For a reproducible target, test against any site that uses CloudFlare, for example https://yts.im

@indutny
Node.js Foundation member

@xeoncore could you please open a separate issue for this?

@ghost

@indutny Its not really a issue for me, I just thought I'd leave the info here considering a few comments up someone asked

Can you explain/link to the changes google made that caused this? Presumably if it mattered which frontend we hit, this bug was triggered by something on your end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment