0.10.x https/ssl and ECONNRESET error (part 2)

[cybo42]

I am having an almost identical issue as reported in issue #5119. I could not figure out how to re-open the issue.

I am trying to make a call out to a HTTPS RESTful endpoint and receiving the below error:

HTTP: outgoing message end.
HTTP: HTTP SOCKET ERROR: read ECONNRESET
Error: read ECONNRESET
    at errnoException (net.js:884:11)
    at TCP.onread (net.js:539:19)
problem with request: read ECONNRESET
HTTP: HTTP socket close

I am running v0.10.5. When I run this under v0.8.18 everything works fine.

Originally I was using the request node module, however for debugging purposes I switch to https.

I have even copied the sample code from #5119 and updated the host and path to my endpoint and still received the same error.

The certificate is NOT a self-signed certificate.

Any insight would be helpful, and please let me know if any additional information can be used to help debug.

[bnoordhuis]

ECONNRESET means the other end unexpectedly closed the connection but that's the 'what', not the 'why'. Can you post a test case?

[cybo42]

I don't have a true "test case" but below is a quick test script I was using to verify. I also created a second test script based on the test code in #5119 just updating it with my hostname and removing unused arguments.

Original test code:
I've tried variations of leaving out strictSSL, and rejectUnauthroized with same results

// request version 2.20.0
var request = require('request');

var opts = {
    url: 'https://user-service.condenastdigital.com/'
    , method: 'GET'
    , strictSSL: false
    , rejectUnauthorized: false
    ,  agent: false
};

request(opts, function(error, response, rawResponse){
    if(error) throw error;

    if(200 === response.statusCode){
        console.log("Headers", JSON.stringify(response.headers, null, 2));
        console.log("GOOD Response:\n");
        console.log(rawResponse);
    }else{
        console.log("Headers", JSON.stringify(response.headers, null, 2));
        console.log("Http Status: " + response.statusCode);
        console.log("Unexpected Response:\n");
        console.log(rawResponse);

    }
});

Recreated test code based on issue #5119

var https = require('https');

var headers = {};

var options = {
  host : 'user-service.condenastdigital.com',
  port: 443,
  path : '/',
  method : 'GET',
  headers : headers,
  agent: false,
  secureOptions: require('constants').SSL_OP_NO_TLSv1_2

};

function callback(res) {
  console.log('STATUS: ' + res.statusCode);
  console.log('HEADERS: ' + JSON.stringify(res.headers));
  res.setEncoding('utf8');
  var str = '';
  res.on('data', function (chunk) {
    str += chunk;
  });
  res.on('end', function () {
    console.log(str);
  });
}

var req = https.request(options, callback);
req.on('error', function(e) {
  throw e;
});

req.end();

Both code snippets work fine under node v0.8.18 and fail under v0.10.5 with same error

Error: read ECONNRESET
    at errnoException (net.js:878:11)
    at TCP.onread (net.js:539:19)

[svmehta]

I'm also seeing a sharp uptick in network errors since upgrading to node 0.10.5 from 0.8.22. (ECONNRESET, ETIMEDOUT, etc). These are always bubbling up as uncaught exceptions - is there a change between 0.8 and 0.10 that would cause this?

2013-04-26T03:48:49.637Z - error: uncaughtException: stack=Error: read ECONNRESET
at errnoException (net.js:884:11)
at TCP.onread (net.js:539:19), message=read ECONNRESET, stacktrace=
at process. (...)
at process.EventEmitter.emit (events.js:95:17)
at process._fatalException (node.js:272:26)

[svmehta]

Ok, I actually see the changelog to 0.9.10 indicates that these errors are no longer suppressed... I guess we can just suppress it ourselves since when dealing with 1000s of network requests a second some are bound to timeout, reset by peer, etc. Is there a suggested way to handle such issues? Why were they originally suppressed by node itself?

[bnoordhuis]

Thanks, I can reproduce the issue. Reduced test case:

require('https').get('https://user-service.condenastdigital.com/', function(res) {
  res.pipe(process.stdout);
});

Or as a tls module test case:

require('tls').connect(443, 'user-service.condenastdigital.com', function() {
  this.write('GET / HTTP/1.0\r\nHost: user-service.condenastdigital.com\r\n\r\n');
  this.pipe(process.stdout);
});

In both cases, the connection simply hangs before any cleartext gets exchanged. Debug logging suggests the connection stalls after (or maybe during) the TLS handshake:

TLS: encrypted.read called with 16384 bytes
TLS: encrypted.read succeed with 0 bytes
NET: 99264 connect: find host user-service.condenastdigital.com
TLS: encrypted.read called with 16384 bytes
TLS: encrypted.read succeed with 362 bytes
TLS: encrypted.read called with 16384 bytes
TLS: encrypted.read succeed with 0 bytes
NET: 99264 _read
NET: 99264 _read wait for connection
TLS: encrypted.read called with 16384 bytes
TLS: encrypted.read succeed with 0 bytes
NET: 99264 afterConnect
NET: 99264 _read
NET: 99264 Socket._read readStart
NET: 99264 afterWrite 0 { bytes: 362, oncomplete: [Function: afterWrite] }
NET: 99264 afterWrite call cb

But e.g. encrypted.google.com:443 works.

[bnoordhuis]

Is there a suggested way to handle such issues?

That all depends on your application. There is no canonical approach to dealing with network errors.

Why were they originally suppressed by node itself?

Because ECONNRESET is not always relevant. If you've already torn down the connection and the peer then sends a RST, emitting it isn't useful - you're already done. Before v0.10, node.js was a little overzealous in that it squelched all ECONNRESET errors instead of just the ones that aren't relevant anymore.

[bnoordhuis]

As a FYI, the issue is more with OpenSSL, not node.js. I have a v0.10 branch here that downgrades the bundled OpenSSL to the version that it's in v0.8 and with that version the test cases work flawlessly.

[cybo42]

@bnoordhuis Thanks for the investigation, I just built node off of your branch, and confirmed now I can access the site w/o issue.

Hopefully this gets merged into v0.10.6.

Any ideas on why this version of OpenSSL causes this issue.

[bnoordhuis]

@cybo42 Thanks for testing. Landed in 4fdb8ac.

Any ideas on why this version of OpenSSL causes this issue.

I'm reasonably sure it's because of the TLS v1.2 support that's enabled by default in v1.0.1 and maybe because of a few new TLS extensions it supports (but whose advertisements are not handled gracefully by all existing servers.)

[niknah]

I had this ECONNRESET error with another site. But the latest nodejs from git didn't fix it for me.

Forcing sslv3 fixed it for me. By adding this into the options...

secureProtocol: "SSLv3_method"

Seems to work for https://user-service.condenastdigital.com/ too.

I also went back to the old versions, it stopped working between node v0.9.1 - v0.9.2

[bnoordhuis]

We were able to establish that the issue lies with user-service.condenastdigital.com and not openssl. Limiting the number of advertised ciphers, disabling SNI, etc. all fix the connection issue. Our current working hypothesis is that it uses some kind of fixed-size buffer for parsing TLS handshake data. As a result, the downgrade has been undone in 2cf7e5d.

Passing { secureOptions: constants.SSL_OP_NO_TLSv1_2 } to tls.connect() or https.request() seems to fix the issues people have reported with IIS.

[isaacs]

@bnoordhuis Thanks!

[cybo42]

So I tried using the { secureOptions: constants.SSL_OP_NO_TLSv1_2 } you specified, however I still received the ECONNRESET error. After some digging I noticed that the user-service.condenastdigital.com SSL cert us not using the same encryption cipher as most of our other servers.

From using curl I see user-service uses RC4-SHA opposed to AES256-SHA for most of our servers.

I did some digging and found if I use these options below, my call works (using both https and tls modules)

var constants = require('constants');

var opts = {
    hostname: 'user-service.condenastdigital.com'
    , port: 443
    , secureOptions: constants.SSL_OP_NO_TLSv1_2
    , ciphers: 'ECDHE-RSA-AES256-SHA:AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM'
    , honorCipherOrder: true

};

require('https').get(opts, function(res) {
  res.pipe(process.stdout);
});

tls

var constants = require('constants');
var tlsOpts = {
    secureOptions: constants.SSL_OP_NO_TLSv1_2
    , ciphers: 'ECDHE-RSA-AES256-SHA:AES256-SHA:RC4-SHA:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM'
    , honorCipherOrder: true
};


require('tls').connect(443, 'user-service.condenastdigital.com', tlsOpts, function() {
  this.write('GET / HTTP/1.0\r\nHost: user-service.condenastdigital.com\r\n\r\n');
  this.pipe(process.stdout);
});

So appears that you are correct that its not a issue with Node per say, just user-service needs some additional configuration to talk to it. I though that I saw somewhere that the preferred cipher list was changed or updated in the v0.10.x release but I can't seem to find the link/note right now.

I'd love any insight if someone could explain the need for these configuration values.

Now I can successfully use the https module to access the server. Now I am just trying to see how to get the request module to respect the same options.

[bnoordhuis]

So I tried using the { secureOptions: constants.SSL_OP_NO_TLSv1_2 } you specified, however I still received the ECONNRESET error.

If all else fails, you can always use SSL_OP_NO_TLSv1 (which is an inverted way of saying 'only SSLv2 and 3.')

I though that I saw somewhere that the preferred cipher list was changed or updated in the v0.10.x release but I can't seem to find the link/note right now.

That happened some time ago in the v0.8 series, see commit badbd1a. However, v0.8 ships with a version of openssl that speaks only TLS v1.0 by default so only a subset of the ciphers are used.

[realyze]

+1 for fixing this.

I'm seeing this issue with e.g. 'https://conseq.cz/' - I'm getting { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }. Using superagent or request modules, no special options.

Installing node0.8 via nvm "fixes" the issue.

[jsdevel]

Thanks guys! I was able to get this to work using 10.20. Here's what I used:

var constants = require('constants');
var fs = require('fs');
var https = require('https');
var path = require('path');

var PFX = path.resolve(__dirname, 'mylocal.pfx');

var options = {
  headers:{
    "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/
webp,*/*;q=0.8",
    "Accept-Encoding":"gzip,deflate,sdch",
    "Accept-Language":"en-US,en;q=0.8",
    "Cache-Control":"no-cache",
    "Connection":"keep-alive",
    "Pragma":"no-cache",
    "User-Agent":"Mozilla/5.0 (Linux; U; Android 4.0.2; en-us; Galaxy Nexus
Build/ICL53F) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/
534.30"
  },
  hostname:"iis.host",
  path: '/SomeService/Service.asmx?wsdl',
  method: 'GET',
  pfx:fs.readFileSync(PFX),
  passphrase:"passphrase",
  agent:false,
  rejectUnauthorized:false,
  secureOptions: constants.SSL_OP_NO_TLSv1_2,
  strictSSL:false
};

var req = https.request(options, function(res){
  console.dir(res);
  res.pipe(process.stdout);
});

req.on('error', function(){
  console.log("============");
  console.dir(arguments);
});

req.end();

[garthk]

Note for those arriving here thinking 0.12 hosed them: if secureOptions: constants.SSL_OP_NO_TLSv1_2 in the options to https.request eliminate the socket hangup, the root cause might be that the old values you had for key and cert were too short or too stale.