Skip to content
This repository

making requests to https://google.com errors #4771

Closed
azylman opened this Issue · 37 comments

15 participants

Alex Zylman Ben Noordhuis Rodrigo Alves Alexey Kupershtokh Cory Wilkerson abarron87 Adam Langley Alexander Turner Isaac Z. Schlueter José F. Romaniello angelochen960 Ben Buckman Zane Claes Michał Mańko Aaron Dufour
Alex Zylman

This happens for me in node v0.8.12 and v0.8.19 on two different machines, one Ubuntu 12.04 and one OSX 10.8. It doesn't happen if I make requests to other domains.

var https = require('https');
https.request('https://google.com', function(res) {console.log(res)});

Produces:

events.js:68
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140463203215168:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:656:24)
    at SlabBuffer.use (tls.js:199:18)
    at CleartextStream.CryptoStream._push (tls.js:483:33)
    at SecurePair.cycle (tls.js:880:20)
    at SecurePair.cycle (tls.js:895:10)
    at EncryptedStream.CryptoStream.write (tls.js:267:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:93:17)
    at TCP.onread (net.js:396:14)
Ben Noordhuis

Sorry, can't reproduce. Works for me with master and v0.8.

Rodrigo Alves

@azylman Couldn't that be a problem with your local openssl program? Have you tested it for other HTTPS urls?

Tested here on Mac OSX Mountain Lion and everything is fine.

Alex Zylman

@rodrigoalvesvieira I did test it with other HTTPS URLs and it only failed hitting Google. I'm not sure how it could be a problem with my local openssl program when I tried it on two different machines with two different versions of OpenSSL. Their versions of openssl were: OpenSSL 0.9.8r 8 Feb 2011 and OpenSSL 1.0.1 14 Mar 2012.

I asked one of my coworkers to confirm and he sees the same behavior, so this is not an isolated incident...

Ben Noordhuis

Different machines but the same network? Check that there's not some kind of firewall or proxy causing interference.

Alex Zylman

I've reproduced it on three different networks so far. On an AWS machine, on our office network, and also while tethering on Verizon's 4G network.

Alexey Kupershtokh

I have the same issue when trying to work with google under xubuntu 12.10 and nodejs v0.8.19-0.8.20.

events.js:71
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140423179986752:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at SecurePair.cycle (tls.js:913:10)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)
Cory Wilkerson

Seeing this on Mountain Lion 10.8.2 and nodejs v0.8.20

> var options = { 
    hostname: 'gdata.youtube.com', 
    port: 443, 
    path:'/feeds/api/playlists/snippets?q=GoogleDevelopers', 
    headers: { 'User-Agent':'youtube-feeds.js (https://github.com/fvdm/nodejs-youtube)', 'Accept': 'application/json' }, 
    method: 'GET' 
}

> var request = https.request( options, function(res) { console.log(res) });

events.js:71
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: 140735290048896:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at SecurePair.cycle (tls.js:913:10)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)
Alexey Kupershtokh

And Ubuntu 11.04 + node 0.8.19-1chl1~natty1 and 0.8.20-1chl1~natty1 (i386, from chris lea's ppa) on my office pc has the same issue in both cases from @corywilkerson and @azylman

Though Ubuntu 10.04.4 LTS + node v0.8.16-1chl1~lucid1 and 0.8.20-1chl1~lucid1 (amd64, also from chris lea's ppa) work well on a server located in a DC.

Alexey Kupershtokh

Reopen maybe?

Ben Noordhuis

For some reason it reproduces for me now (with google.com - but not www.google.com).

Alexey Kupershtokh

I've also noticed this. This could probably be related:

wicked@wicked-desktop:/$ wget -S https://google.com/
--2013-02-19 18:46:12--  https://google.com/
Resolving google.com... 74.125.143.138, 74.125.143.139, 74.125.143.100, ...
Connecting to google.com|74.125.143.138|:443... connected.
ERROR: certificate common name `*.google.com' doesn't match requested host name `google.com'.
To connect to google.com insecurely, use `--no-check-certificate'.

But this does not explain why does node still crash on an https://play.google.com/ url.

abarron87

I am getting the same issue with the YouTube data API, it worked yesterday afternoon but coming in the morning I see:

Error: 140735247131008:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288

Running node 0.8.15

Hostname: "gdata.youtube.com"
Path: "/feeds/api/videos?q=arsenal&orderby=relevance&v=2&alt=json&max-results=2&key="+apiKey

Requests work in the browser.

Adam Langley

Reproducibility for this issue will vary because of differing configs across our frontend clusters and for different domains. However, over time, I expect increasing amounts of Google and non-Google services to stop working because of this.

The wget issue noted above is due to a deficiency in wget (lack of SNI support) and is unrelated.

I think the problem arises from a failure to clear the OpenSSL error stack in node_crypto.cc:

EVP_PKEY *pkey = NULL;
RSA *rsa = NULL;
if( NULL != (pkey = X509_get_pubkey(peer_cert))
    && NULL != (rsa = EVP_PKEY_get1_RSA(pkey)) ) {
    BN_print(bio, rsa->n);

When EVP_PKEY_get1_RSA fails it will leave an error on the stack and that appears to be popping up later in processing.

I cannot reproduce this on master and I bisected the fix to d59beb9. However, that's a wider ranging change that likely fixed the issue by accident or, possibly, masked it for a little longer.

As a smaller change that can be back-ported, I'd suggest checking that the public key is an RSA key before calling EVP_PKEY_get1_RSA:

EVP_PKEY *pkey = NULL;
RSA *rsa = NULL;
if( NULL != (pkey = X509_get_pubkey(peer_cert))
    && EVP_PKEY_RSA == EVP_PKEY_id(pkey)
    && NULL != (rsa = EVP_PKEY_get1_RSA(pkey)) ) {
    BN_print(bio, rsa->n);

(Note the extra clause in the condition.)

That fixes the proximal issue for me although the problem of not cleaning the OpenSSL error stack appears to be more pervasive.

Luke Bayes lukebayes referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Luke Bayes lukebayes referenced this issue from a commit in lukebayes/node
Luke Bayes Fixed unexpected non-RSA verification failures.
This fix ensures that the RSA verification is only used for RSA signatures.

joyent/node#4771
e6ce416
Alexander Turner

Still having this issue on v0.8.21, currently pushing to google over HTTP, though HTTPS is still failing with:
Error: 140692638627648:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

Isaac Z. Schlueter
Collaborator

@bnoordhuis Any idea when you'll get a chance to look into this? I think it's causing some occasional errors talking to the npm registry as well.

Ben Noordhuis

I have a preliminary patch but it seems to slightly regress make bench-tls. That's on a MBA though so take with some salt.

diff --git a/src/node_crypto.cc b/src/node_crypto.cc
index 187b344..44d2171 100644
--- a/src/node_crypto.cc
+++ b/src/node_crypto.cc
@@ -901,6 +901,16 @@ int Connection::HandleBIOError(BIO *bio, const char* func, int rv) {


 int Connection::HandleSSLError(const char* func, int rv, ZeroStatus zs) {
+  // Forcibly clear OpenSSL's error stack on return. This stops stale errors
+  // from popping up later in the lifecycle of the SSL connection where they
+  // would cause spurious failures. It's a rather blunt method, though.
+  // ERR_clear_error() isn't necessarily cheap either.
+  struct ClearErrorOnReturn {
+    ~ClearErrorOnReturn() { ERR_clear_error(); }
+  };
+  ClearErrorOnReturn clear_error_on_return;
+  (void) &clear_error_on_return;  // Silence unused variable warning.
+
   if (rv > 0) return rv;
   if ((rv == 0) && (zs == kZeroIsNotAnError)) return rv;

Ben Noordhuis bnoordhuis closed this issue from a commit
Ben Noordhuis bnoordhuis crypto: clear error stack
Clear OpenSSL's error stack on return from Connection::HandleSSLError().
This stops stale errors from popping up later in the lifecycle of the
SSL connection where they would cause spurious failures.

This commit causes a 1-2% performance regression on `make bench-tls`.
We'll address that in follow-up commits if possible but let's ensure
correctness first.

Fixes #4771.
c6e2db2
José F. Romaniello

is there a 0.8.x not affected by this issue? I don't know what happen, maybe google changed something but I started to see this error randomly few days ago and I can't find a version not affected.

I am not sure how to workaround it

If not, any ETA of a new version with the fix?

thanks a bunch

Ben Noordhuis bnoordhuis referenced this issue from a commit in bnoordhuis/node
Ben Noordhuis bnoordhuis crypto: clear error stack
Clear OpenSSL's error stack on return from Connection::HandleSSLError().
This stops stale errors from popping up later in the lifecycle of the
SSL connection where they would cause spurious failures.

This commit causes a 1-2% performance regression on `make bench-tls`.
We'll address that in follow-up commits if possible but let's ensure
correctness first.

Fixes #4771.

This is a back-port of commit c6e2db2 from the master branch.

Conflicts:
	src/node_crypto.cc
1e7b3d5
Ben Noordhuis

The patch hasn't been back-ported to v0.8 yet, it needs a little more testing for unforeseen side effects. The back-port commit is bnoordhuis/node@1e7b3d5 in case you want to try it out.

José F. Romaniello

Thanks, I will try it. I have been testing my use case with all 0.8.x releases and 0.8.1, 0.8.2, and 0.8.3 seems to not have this problem. Version 0.8.4 to 0.8.21 (included) fails.

The url I am using in my application is https://apps-apis.google.com/a/feeds...

Ben Noordhuis

Hm, that would suggest that eb2ca10 is the culprit. Can you connect when you set { rejectUnauthorized: false } in the options object?

José F. Romaniello

I tried with { rejectUnauthorized: false } and still same issue:

[Error: 1077155136:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288

I went into tls.js, the validation and the regex (changed in that commit you point me) seems to work ok..

Ben Noordhuis

Okay, in that case the back-ported patch should fix it for you.

angelochen960

in OS X mountain lion, Node 0.8.18 and 0.8.21, a query to youtube api, sometimes it works, sometimes, got following error:

https://gdata.youtube.com/feeds/api/videos/wzzUjm8apzA?v=2&alt=json
[Error: 140735083155840:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

any idea how to fix this? Thanks,

Ben Noordhuis

Have you tried the patch? (Hint: "no" is not an acceptable answer. "Yes, but it doesn't work" however is.)

angelochen960

sorry, I did not, how to apply the patch? in OS X, I just run the standard installer from dmg.

Ben Noordhuis

Try this:

$ git clone git://github.com/joyent/node.git && cd node
$ git checkout origin/v0.8
$ curl -s https://github.com/bnoordhuis/node/commit/1e7b3d5.patch | git am
$ ./configure && make

The binary ends up in out/Release, you can install it with sudo make install.

Ben Noordhuis

Oh, and add -j<cores> if you want to speed up the build. Replace <cores> with the number of CPUs in your machine.

angelochen960

thanks, tried the patch, it works, then again I went back to 0.8.21, it works as well, I have to try some more to see if 0.8.21 will have that error again, then I will re-try the patch, looks like an intermittent error.

Ben Buckman

After installing 0.8.22 from source with the 1e7b3d5 cherry-picked, I'm still getting errors:

Error: 140735117736288:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:683:

at CleartextStream._puller (tls.js:667:24)
at CleartextStream.CryptoStream._pull (tls.js:601:19)
at SecurePair.cycle (tls.js:891:20)
at EncryptedStream.CryptoStream.write (tls.js:285:13)
at Socket.ondata (stream.js:38:26)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:397:14)

when I request('https://google.com'), request('https://yahoo.com'), or my personal site on https.

(On OSX 10.7.5)

Zane Claes

+1

Ben Noordhuis

@newleafdigital If that's the request method from the request module from npm, please open a bug report there. If it's the request method from the built-in http module, please post a test case because I can't reproduce it.

Michał Mańko

Hey, I have the same problem, simple test case:

var https = require('https');

var url = 'https://www.youtube.com/watch?v=2I0P_Dpd5xo'; // doesn't work
// url = 'https://google.com'; // doesn't work
// url = 'https://www.facebook.com'; // it works
// url = 'https://github.com'; // it works
// url = 'https://login.yahoo.com/'; // it works
var href = require('url').parse(url);

var request = https.request(href, function(res) {
        console.log("statusCode: ", res.statusCode);
        console.log("headers: ", res.headers);
        res.on('data', function(d) {
                process.stdout.write(d);
        });
});
request.end();
request.on('error', function(e) {
        throw e;
});

and the exception:

nodeHttpsError.js:19
        throw e;
              ^
Error: 139698244785984:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:

    at CleartextStream._pusher (tls.js:674:24)
    at SlabBuffer.use (tls.js:217:18)
    at CleartextStream.CryptoStream._push (tls.js:501:33)
    at SecurePair.cycle (tls.js:898:20)
    at EncryptedStream.CryptoStream.write (tls.js:285:13)
    at Socket.ondata (stream.js:38:26)
    at Socket.EventEmitter.emit (events.js:96:17)
    at TCP.onread (net.js:397:14)

I tested it on several environments:

  • Ubuntu 12.04.2 + Node 0.8.22 + OpenSSL 1.0.1 - Network A - doesn't work
  • Ubuntu 12.04.2 + Node 0.8.21 + OpenSSL 1.0.1 - Network A - doesn't work
  • Ubuntu 10.04.4 + Node 0.8.20 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 10.04.4 + Node 0.8.18 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 10.04.4 + Node 0.8.22 + OpenSSL 0.9.8k - Network B - works
  • Ubuntu 12.04.2 + Node 0.8.16 + OpenSSL 1.0.1 - Network B - works

It looks strange, in my case it could be one of two problems:

  • A bug is in new Node versions (>0.8.16) and only on Ubuntu 12
  • Network configuration is the problem (?)

I will try to upgrade last machine to the newest Node version and check it out, but not today.

If you need more info, feel free to ask.

Update:
One more server:

  • Ubuntu 12.04.2 + Node 0.8.22 + OpenSSL 1.0.1 - Network B - works
Ben Noordhuis

I can confirm that the issue exists with v0.8 but not v0.10 and that the proposed patch or upgrading openssl doesn't fix it.

Interestingly enough, git bisect pinpoints c6e2db2 as the commit that fixes it.

Ben Noordhuis

upgrading openssl doesn't fix it

However, applying the patch and linking to the 0.98 system openssl does fix it. Hm.

Ben Noordhuis bnoordhuis reopened this
Ben Noordhuis

It seems the bug is in (or triggered by) lib/http.js or lib/https.js because a plain TLS connection works. That is, the following test case works:

var tls = require('tls');
var conn = tls.connect(443, 'www.youtube.com', function() {
  conn.pipe(process.stdout);
  conn.write('GET /watch?v=2I0P_Dpd5xo HTTP/1.1\r\n' +
             'Host: www.youtube.com\r\n' +
             '\r\n');
});
Ben Noordhuis bnoordhuis referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Ben Noordhuis bnoordhuis referenced this issue from a commit
Ben Noordhuis bnoordhuis crypto: check key type in GetPeerCertificate()
Works around the following exception:

  Error: 140463203215168:error:0607907F:digital envelope
  routines:EVP_PKEY_get1_RSA:expecting an rsa key:
  ../deps/openssl/openssl/crypto/evp/p_lib.c:288:
    at CleartextStream._pusher (tls.js:656:24)
    at SlabBuffer.use (tls.js:199:18)
    at CleartextStream.CryptoStream._push (tls.js:483:33)
    at SecurePair.cycle (tls.js:880:20)
    <snip>

The issue has been solved properly in v0.10 and the master branch as of
commit c6e2db2 ("crypto: clear error stack"). This is the "back-port"
to v0.8.

For some (rather unquantifiable) reason the original fix only works for
the tls module in v0.8 but not the https module unless OpenSSL is
downgraded to 0.9.8. Upgrading OpenSSL does *not* fix it, however.

The https module doesn't appear to be at fault; upgrading it to v0.10
doesn't fix the issue. That leaves either the tls or the http module
(that https derives from) but the changes to those modules are too
massive to back-port as-is.

`git bisect` over the v0.8 -> v0.10 commits didn't show up anything
useful, it pinpoints c6e2db2 as the commit that fixes things.

I've spent several hours on this now and seeing that v0.8 is in
maintenance mode, this cheap hack will have to do.

Fixes #4771.
2c41a80
Ben Noordhuis

'Fixed' in v0.8 per 2c41a80. I've tried a lot of things but ended up settling for @agl's hack...

Franklin van de Meent fvdm referenced this issue from a commit in fvdm/nodejs-youtube
Franklin van de Meent fvdm Replaced HTTPS with HTTP to fix `connection error`
This module does not handle user credentials.

> Error: 140735110496640:error:0607907F:digital envelope routines:EVP_PKEY_get1_RSA:expecting an rsa key:../deps/openssl/openssl/crypto/evp/p_lib.c:288:
1e34663
Franklin van de Meent fvdm referenced this issue in fvdm/nodejs-youtube
Closed

Update npm package to reflect recent bug fixes #3

Franklin van de Meent fvdm referenced this issue from a commit in fvdm/nodejs-youtube
Franklin van de Meent fvdm Avoiding Node 0.8.4-0.8.22 due to SSL bug (#3)
Node versions 0.8.4-0.8.22 appear to be affected by this bug, therefore the module is now dependent on earlier and later versions.

Refer to ticket joyent/node#4771 for details.
9a4dad1
Mina Mikhail fightingtheboss referenced this issue in meteor/meteor
Closed

SSL bug in Node 0.8.4 - 0.8.22 #859

Aaron Dufour

@agl Can you explain/link to the changes google made that caused this? Presumably if it mattered which frontend we hit, this bug was triggered by something on your end.

Justin Parker jparkrr referenced this issue in Singly/hallway
Merged

Temp gdata http #861

Dane Springmeyer springmeyer referenced this issue in mapbox/millstone
Closed

test failures with node v0.8.22 #98

Sam samcday referenced this issue in samcday/node-google-spreadsheets
Open

Fixed SSL #9

bowdev bowdev referenced this issue from a commit in bowdev/arrow
aresyhoo Provide default port 443 when do https request
Due to a bug in nodejs Node 0.8.4 - 0.8.22, according to
joyent/node#4771

We need add port 443 as default in our own https tests.
7900b30
bowdev bowdev referenced this issue from a commit in bowdev/arrow
aresyhoo Support https port 443
To workaround a SSL bug in https module for node 0.8.4-0.8.22 as
mentioned at joyent/node#4771
475e5d6
bowdev bowdev referenced this issue in yahoo/arrow
Merged

Support https port 443 #152

Grant J. Butler grantjbutler referenced this issue from a commit in grantjbutler/spikebot
Grant J. Butler grantjbutler Fixed some potential HTTPS request issues outlined here: joyent/node#…
…4771 . Fix one last formatting thing for time. Dang it, JavaScript
72370a1
jerome creignou jcreigno referenced this issue in jcreigno/nodejs-mail-notifier
Closed

Error event doesn't seem to work #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.