App throwing ECONNRESET multiple times

[manjufy]

• Node.js Version: v8.12.0
• V8: 6.2.414.66
• OS: MacOS x64 Darwin
• Scope (install, code, runtime, meta, other?): Serving Node using nginx and Phusion Passenger
• Module (and version) (if relevant): net

Keep seeing the following error nodejs api

Error: read ECONNRESET
    at _errnoException (util.js:992:11)
    at TCP.onread (net.js:618:25)
  code: 'ECONNRESET',
  errno: 'ECONNRESET',
  syscall: 'read',
  fatal: true

[manjufy]

I upgraded node version from 8.12.0 to 10.19.0

Now I get a different error

events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Emitted 'error' event at:
    at Socket.socketErrorListener (_http_client.js:401:9)
    at Socket.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

[manjufy]

I managed to reproduce the above issue following this post nodejs/node#27916

[FractalHippy]

Yes, I can reproduce this too on both 12.16.1 and 13.12.0, and it's definitely nodejs/node#27916.

[manjufy]

OK, one of the issue I was facing was using knex pool connection.

This was the connection object

var knex = require('knex')({
  client: 'mysql',
  connection: {
    host : '127.0.0.1',
    user : 'your_database_user',
    password : 'your_database_password',
    database : 'myapp_test'
  },
});

However, knex pool connection uses the default settings i.e. min: 2, max: 10

Since knex always keeps 2 connections open, these 2 connections may not necessarily be active in MySQL. When an application makes a connection to DB using one of those stale connection MySQL rejects these connections hence, the ECONNRESET error on

Error: read ECONNRESET
    at _errnoException (util.js:992:11)
    at TCP.onread (net.js:618:25)
  code: 'ECONNRESET',
  errno: 'ECONNRESET',
  syscall: 'read',
  fatal: true

Notice that the above error is raised from .net library not from HTTP client.

I managed to solve this issue by having the following config

var knex = require('knex')({
  client: 'mysql',
  connection: {
    host : '127.0.0.1',
    user : 'your_database_user',
    password : 'your_database_password',
    database : 'myapp_test'
  },
  pool: { min: 0, max: 7 }
});

keep the pool min: 0 always.
That is to say, if there are no active db pool connections, just don't create any pool connections. Keeping min: 2, will just make application to keep 2 stale connections if there are no active activity on your application.

However, that just solved on the problem.
Now I have the error on a different problem.

error: Uncaught Exception: { Error: socket hang up
    at createHangUpError (_http_client.js:332:15)
    at Socket.socketCloseListener (_http_client.js:373:25)
    at Socket.emit (events.js:203:15)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at TCP._handle.close (net.js:607:12) code: 'ECONNRESET' }

Now the above error is rising from an HTTP client, not a network issue any more. not sure how to solve this yet.

[hankdesu]

Hi @manjufy,
I got issue when I use Knex to do MySQL connection, I set connection pool min to 1, and I randomly got an error like Connection Error: Error: Connection lost: The server closed the connection.
You mentioned that if I set the minimum connection pool Knex will always keeps connections open, and MySQL will reject that stale connection.
You mean that MySQL has already delete that stale connection, but Knex still use it to make a connection?
So if I want to reproduce this error case, I can set connection pool max and min to 1. When I use Knex to create a connection, I kill that connection by DB tool, and I query DB again, it will throw these error, am I right?
Thanks for spent time reading my comment, if you could tell me above understanding is correct or not, That would be great!😃

[manjufy]

@hankdesu Yes that's right.

You mean that MySQL has already deleted that stale connection, but Knex still use it to make a connection? => Yes, that's correct. However, it depends on your servers. In my case, there was always a connection lost between API Server and MySQL server (They are both different physical servers). So when you have min pool in knext set to 1, In your API server, there is always one connection open to MySQL, On the other hand, MySQL may not have that connection active in its pool. So when your app/api with knex call MySQL, these knex pool connection may be stale.

The server closed the connection could be something else too. Check your MySQL timeout connection in the MySQL config. If you are getting this too often it could be timeout either in MySQL or knex connection settings

[hankdesu]

@manjufy Very grateful for your response!😃
After a few days observe, In one of mine production environment, I found out that when Knex finish query, it did not release the connection, Therefore the whole DB operation stuck until timeout, But when I modify connection pool minimum to 0, Every time Knex finish query, it will release the connection.
It's weird, but it work

[manjufy]

@hankdesu Glad, you figured that and it worked out 👏🏿 .
That's exactly what I did, tweaked knex pool connection to 0 and observed over a period of time and the problem stopped occurring.