Potential SSL/TLS issue when remapping

[nosmo]

I believe there is an SSL problem somewhere in Bundler's remapping - despite extensive debugging, I can't figure out where. When not using remaps, there are zero issues with any of the domains in question.

In my branch (#4) I have added deflect.ca as an origin for distributed.deflect.ca, in addition to fulltimeinter.net as an origin for nosmo.me. The latter works just fine, but the former fails to remap as follows:

debug: in Bundler.send/async.reduce, memo = url=http://distributed.deflect.ca, strictSSL=false, agent=false, rejectUnauthorized=false
debug: in Bundler.send/async.reduce, memo = url=http://distributed.deflect.ca, strictSSL=false, agent=false, rejectUnauthorized=false,
debug: in Bundler.send/async.reduce, memo = url=http://distributed.deflect.ca, strictSSL=false, agent=false, rejectUnauthorized=false, Origin=Bundler
debug: in Bundler.send/async.reduce, memo = url=http://distributed.deflect.ca, strictSSL=false, agent=false, rejectUnauthorized=false, Origin=Bundler, followRedirect=true, followAllRedirects=false, maxRedirects=10
Remapping URL to http://deflect.ca/. Hostname is distributed.deflect.ca
error: Error making request to http://distributed.deflect.ca; Error: Error: socket hang up
    at createHangUpError (http.js:1472:15)
    at Socket.socketOnEnd [as onend] (http.js:1568:23)
    at Socket.g (events.js:180:16)
    at Socket.emit (events.js:117:20)
    at _stream_readable.js:929:16
    at process._tickCallback (node.js:419:13) socket hang up
Failed to create bundle for /?url=http://distributed.deflect.ca
Error: socket hang up

Deflect hosts' SSL setups are not exactly unique but they have some specifics - most obviously disabling SSLv2 and SSLv3 completely. There is a restricted but still quite generous cipher list in use also. In this particular case deflect.ca redirects to https://deflect.ca. The HTTPS redirect itself is not an issue as I've tested this in other places.

It looks like this could be related to nodejs/node-v0.x-archive/issues/5360, but I'm not certain. Signs definitely point that way, as using other sites that claimed to be experiencing similar issues as a remap also produces this error.

Getting this with node v0.10.33 and v0.12.0.

[nosmo]

The plot sickens - I get this on a Debian box with our own build of v0.12.0:

Error: Error: write EPROTO 121631263627040:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:770:

    at exports._errnoException (util.js:746:11)
    at WriteWrap.afterWrite (net.js:766:14) write EPROTO 121631263627040:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:770:

Failed to create bundle for /?url=http%3A%2F%2Fdistributed.deflect.ca%2F
Error: write EPROTO 121631263627040:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/s23_clnt.c:770:

[willdoran]

Can we map this stage by stage? Is it currently configed to use tls?

[nosmo]

I'm not sure - to be honest I can't quite figure out how to force TLS instead of optionally falling back to SSLv3. There are constants that can be used but the documentation is quite poor

[arcrose]

In nodejs/node-v0.x-archive#5360 the solution seems to be to force SSLv3 but that is obviously not something we would want to do on every request.

Passing { secureOptions: constants.SSL_OP_NO_TLSv1_2 } to tls.connect() or https.request() seems to fix the issues people have reported with IIS.

We can do this with the request library by proving the agentOptions field to the options argument to requests. E.g.

request({ url: 'https://google.com', agentOptions: { secureOptions: constants.SSL_OP_NO_TLSv1_2 }}, callback);

I got a list of all TLS and SSL-related constants by requiring the constants module:

{
  SSL_OP_ALL: 2147486719,
  SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION: 262144,
  SSL_OP_CIPHER_SERVER_PREFERENCE: 4194304,
  SSL_OP_CISCO_ANYCONNECT: 32768,
  SSL_OP_COOKIE_EXCHANGE: 8192,
  SSL_OP_CRYPTOPRO_TLSEXT_BUG: 2147483648,
  SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS: 2048,
  SSL_OP_EPHEMERAL_RSA: 2097152,
  SSL_OP_LEGACY_SERVER_CONNECT: 4,
  SSL_OP_MICROSOFT_BIG_SSLV3_BUFFER: 32,
  SSL_OP_MICROSOFT_SESS_ID_BUG: 1,
  SSL_OP_MSIE_SSLV2_RSA_PADDING: 0,
  SSL_OP_NETSCAPE_CA_DN_BUG: 536870912,
  SSL_OP_NETSCAPE_CHALLENGE_BUG: 2,
  SSL_OP_NETSCAPE_DEMO_CIPHER_CHANGE_BUG: 1073741824,
  SSL_OP_NETSCAPE_REUSE_CIPHER_CHANGE_BUG: 8,
  SSL_OP_NO_COMPRESSION: 131072,
  SSL_OP_NO_QUERY_MTU: 4096,
  SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION: 65536,
  SSL_OP_NO_SSLv2: 16777216,
  SSL_OP_NO_SSLv3: 33554432,
  SSL_OP_NO_TICKET: 16384,
  SSL_OP_NO_TLSv1: 67108864,
  SSL_OP_NO_TLSv1_1: 268435456,
  SSL_OP_NO_TLSv1_2: 134217728,
  SSL_OP_PKCS1_CHECK_1: 0,
  SSL_OP_PKCS1_CHECK_2: 0,
  SSL_OP_SINGLE_DH_USE: 1048576,
  SSL_OP_SINGLE_ECDH_USE: 524288,
  SSL_OP_SSLEAY_080_CLIENT_DH_BUG: 128,
  SSL_OP_SSLREF2_REUSE_CERT_TYPE_BUG: 0,
  SSL_OP_TLS_BLOCK_PADDING_BUG: 512,
  SSL_OP_TLS_D5_BUG: 256,
  SSL_OP_TLS_ROLLBACK_BUG: 8388608
}

I could potentially write something that retries requests with one of these options if an error is returned from the request such as the ECONNRESET error described in the bug I referenced earlier. However, turning this into a proper, configurable feature would be quite troublesome and time consuming, so it would be ideal at least for now to think of something that will work as well as we need it to.

There are more options we could provide to agentOptions.

[arcrose]

@nosmo Could you give me precise instructions for how to recreate the error you encountered?

[nosmo]

I reproduced the first error by remapping distributed.deflect.ca to deflect.ca and then running: curl 127.0.0.1:9008/?url=https://distributed.deflect.ca on OS X using node v0.10.33.

I got the error in #5 (comment) by doing the same on a Debian 7 system with node v0.12.0. This error shouldn't be a priority, the first one seems more consistently of concern.

[arcrose]

I've tried all the combinations of the recommendations I found in the node issue referenced earlier and have had no luck with fixing the requests you mentioned. I must confess I'm at a loss for how to handle these issues.
Also see Issue #9 for a related problem.

[nosmo]

I have some good news. This error is maybe possibly almost definitely linked to the options object getting passed here: https://github.com/equalitie/bundler/blob/master/src/bundler.js#L107

This object currently only sets the encoding to null and headers: {Origin: 'Bundler'} - it doesn't set Host (it seems) and it doesn't set any of the SSL flags. By manually hacking in more options mid-function I'm no longer getting these errors (but I am getting some other failure conditions instead).

[arcrose]

I haven't had a lot of time to look into this, but I did insert some logging calls to see what's going on a bit, and I've noticed that when I request distributed.deflect.ca (which is in the remapper), we are getting the socket hangup before reverseProxy is even called to set the Host header. I'll spend some time investigating how it could be crashing so early.

[nosmo]

Just so I don't forget given that we discussed this on IRC - this is probably a problem with Node's cipher list. TLS negotiation begins and then seems to fail before a cipher is agreed upon. Sounds remarkably similar to this issue: http://www.fitter-happier.net/tech/node.js-and-openSSL/. Unfortunately this isn't really a problem that we can expect server operators to fix, rather it is something that needs to be changed in Node. I'll allocate some time to investigate this.

[nosmo]

I think it's safe to close this for now.