Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring. #713

ayazpasha2434 · 2015-02-09T10:25:09Z

Error: Redis connection to redis1.tfs.tfs:6379 failed - read ECONNRESET at RedisClient.on_error (/var/www/a/b/c/node_modules/redis/index.js:196:24) at Socket. (/var/www/a/b/c/node_modules/redis/index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at /var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
at /var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
at process._tickDomainCallback (node.js:486:13)
at process. (/var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/index.js:18:15)

brycebaril · 2015-02-09T18:01:52Z

To me, this sounds like a networking issue outside Node.js, this error "ECONNRESET" is when the remote connection to Node.js closes the socket unexpectedly.

Are you using a proxy between Node and Redis?

What version of Node/node_redis/Redis server?

I doubt the NewRelic agent is involved, but you could always try to reproduce without the agent and see if that fixes the issue. cc/ @wraithan

wraithan · 2015-02-10T01:55:31Z

Those lines are just CLS doing it's thing. With more of a repro I could fully rule out the New Relic agent. But I'm relatively confident this isn't being caused by us.

ayazpasha2434 · 2015-02-10T08:08:27Z

I removed the NewRelic agent, the problem still occurs. Approx. every 3 mins I get this crash happening. Below are the logs for reference.

events.js:72
throw er; // Unhandled 'error' event
^
Error: Redis connection to redis1.tfs.tfs:6379 failed - read ECONNRESET
at RedisClient.on_error (/var/www/a/b/c/node_modules/redis/index.js:196:24)
at Socket. (/var/www/a/b/c/node_modules/redis/index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at process._tickDomainCallback (node.js:486:13)
error: Forever detected script exited with code: 8
error: Script restart attempt #50

Also, I'm using Node.JS - v0.10.35, node_redis - 0.12.1 & Redis - 2.8.4.
I'm accessing Redis directly via IP in Node.

sarathms · 2015-02-12T21:16:50Z

I was facing this problem too, but just got resolved after I realized I was playing around with my /etc/hosts file yesterday for some other reason. It was missing:

127.0.0.1 localhost

@ayazpasha2434 the solution might not be the same for you, but like others are hinting, it is defintely outside node and in the env/shell/OS/VM. Step back a little and stare at it.

ayazpasha2434 · 2015-02-13T07:31:37Z

@sarathms I have this mapping in my hosts file. However, the Redis machine in my case is a remote one. Just cross-checked, I have the said localhost mapping in Redis machine as well. Still the problem is not solved.

Waterloo · 2015-03-08T14:49:31Z

I am also facing the same problem . Here is the Console Output

events.js:72
throw er; // Unhandled 'error' event
^
Error: Redis connection to pub-redis-10606.us-east-1-1.1.ec2.garantiadata.com:10606 failed - read ECONNRESET at RedisClient.on_error (C:\Users\Gigi\Desktop\eduknow_api_ver_1\auth\node_m
odules\redis\index.js:196:24)
at Socket. (C:\Users\Gigi\Desktop\eduknow_api_ver_1\auth\node_modules\redis\index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at process._tickCallback (node.js:442:13)

Waterloo · 2015-03-08T16:39:56Z

@ayazpasha2434 This line solved my error

client.on('error',function(err){ console.error(err)})

blainsmith · 2015-08-15T20:34:21Z

@ayazpasha2434 Did @Waterloo have the right answer for you?

BridgeAR · 2015-09-23T23:15:14Z

There seemed to be an issue with your network connection and it does not seem to be an issue with node redis. Adding the error handler should be sufficient so the app does not crash anymore (and it should always be attached).

biggu0 · 2016-04-07T08:01:51Z

I met this problem when I post a http(use post) to JAVA server.{ [Error: read ECONNRESET]
code: 'ECONNRESET',
errno: 'ECONNRESET',
syscall: 'read',
response: undefined }
Error: read ECONNRESET
at exports._errnoException (util.js:856:11)
at TCP.onread (net.js:546:26)
double callback!
POST /user_archives/create/save 500 45.306 ms - 386
Error: read ECONNRESET
at exports._errnoException (util.js:856:11)
at TCP.onread (net.js:546:26)

BridgeAR · 2016-04-07T12:55:54Z

@biggu0 I'm not sure what connection this has to node_redis. To find out more about ECONNRESET you might want to look at stackoverflow.

akhare22sandeep · 2016-08-02T13:57:36Z

Hi There ,
We are also seeing this issue. We observe that it could be because of redis SNAPSHOTTING . Whenever redis is taking snapshot and storing on the disk connection is getting reset. To prove myself i compare the application logs with redis debug logs and the timings are matching exactly.

below are the snapshotting setting in redis.conf
save 900 1
save 300 10
save 60 10000

Please let us know if we are doing something wrong.

saranyapm · 2017-01-05T15:10:47Z

I am also facing connection issue frequently .

ERR at RedisClient.emit (events.js:188:7)
ERR at RedisClient.on_error (/home/vcap/app/node_modules/redis/index.js:394:14)
ERR at RedisClient. (/home/vcap/app/server.js:28:13)
ERR at emitErrorNT (net.js:1281:8)
ERR at Socket. (/home/vcap/app/node_modules/redis/index.js:272:14)
ERR at process._tickCallback (internal/process/next_tick.js:98:9)
ERR Error: Redis connection to XXXXXXXX:XXXXX failed - read ECONNRESET
ERR at TCP.onread (net.js:572:26)
ERR at _combinedTickCallback (internal/process/next_tick.js:74:11)
ERR at emitOne (events.js:96:13)
ERR at exports._errnoException (util.js:1022:11)
ERR at Socket.emit (events.js:188:7)

After getting the error it is automatically reconnecting.
What can be the issue to lost the connection.

sw-tt-tarunvarma · 2017-01-31T06:22:32Z

Hi Waterloo ,
Where you put this statement client.on('error',function(err){ console.error(err)})
and i think this statement just for console your error nothing else.
is it really usable?
Please give me your thoughts asap.

Thanks in advance!!!

riginoommen · 2017-05-31T06:14:45Z

Hello

I am also facing the same issue

events.js:160
throw er; // Unhandled 'error' event
^

Error: read ECONNRESET

hametGh · 2018-10-04T10:04:34Z

This solved my problem with:

I disabled all of my processes monitoring (new relic, elastic apm ...)
I used this code

process.on('uncaughtException', function (err) {
 console.error(err.stack);
 console.log("Node NOT Exiting...");
});

stormit-vn · 2018-10-22T11:33:44Z

I ma also facing this issue. I am using with sails.js framework with a hook, so there is only one connection to redis server when starting node service, we will use global sails object to access and work with redis

redisClient = redis.createClient({
    host: redisConfig.host,
    port: redisConfig.port,
    db: redisConfig.db,
    retry_strategy: (retryOptions) => {
        if (retryOptions.error && retryOptions.error.code === 'ECONNREFUSED') {
            // End reconnecting on a specific error and flush all commands with
            // a individual error
            return new Error('The server refused the connection');
        }

        if (retryOptions.total_retry_time > 1000 * 60 * 60) {
            // End reconnecting after a specific timeout and flush all commands
            // with a individual error
            return new Error('Retry time exhausted');
        }

        if (retryOptions.attempt > 3) {
            // End reconnecting with built in error
            return undefined;
        }

        sails.log.debug('Handling redis retry strategy', {
            error: retryOptions.error,
            totalRetryTime: retryOptions.total_retry_time,
            attempt: retryOptions.attempt
        });

        // Reconnect after
        return Math.min(retryOptions.attempt * 100, 3000);
    }
});

Here is the stack trace of the error
AbortError: Redis connection lost and command aborted. It might have been processed.\n at RedisClient.flush_and_error (/var/sample-app/node_modules/redis/index.js:362:23)\n at RedisClient.connection_gone (/var/sample-app/node_modules/redis/index.js:664:14)\n at RedisClient.on_error (/var/sample-app/node_modules/redis/index.js:410:10)\n at Socket. (/var/sample-app/node_modules/redis/index.js:279:14)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at emitErrorNT (internal/streams/destroy.js:66:8)\n at /var/sample-app/node_modules/async-listener/glue.js:188:31\n at args.(anonymous function) (/usr/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)\n at _combinedTickCallback (internal/process/next_tick.js:139:11)\n at process._tickDomainCallback (internal/process/next_tick.js:219:9)\n at process.fallback (/var/sample-app/node_modules/async-listener/index.js:563:15).

Our node app is running on a Docker container and it will connect to Redis server installed on an EC2 linux instance.

mallikharjuna-ibexlabs · 2019-04-02T09:17:15Z

We also facing same issue, but here we placed the stunnel to proxy the redis to localhost using below stunnel configuration.

fips = no
#setuid = root
#setgid = root
pid = /var/run/stunnel.pid
debug = 7
#options = NO_SSLv2
options = NO_SSLv3
output = /var/log/stunnel.log
#client = no
#foreground = yes

[redis-cli]
client = yes
accept = 127.0.0.1:6379
connect = redisendpoint:6379

node is disconnecting with these below errors for every couple of minutes. We running these through docker containers.

Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
AbortError: Redis connection lost and command aborted. It might have been processed.

any resolution ?

emps · 2019-05-23T20:40:36Z

We also facing same issue, but here we placed the stunnel to proxy the redis to localhost using below stunnel configuration.

fips = no
#setuid = root
#setgid = root
pid = /var/run/stunnel.pid
debug = 7
#options = NO_SSLv2
options = NO_SSLv3
output = /var/log/stunnel.log
#client = no
#foreground = yes

[redis-cli]
client = yes
accept = 127.0.0.1:6379
connect = redisendpoint:6379

node is disconnecting with these below errors for every couple of minutes. We running these through docker containers.

Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
AbortError: Redis connection lost and command aborted. It might have been processed.

any resolution ?

same to me i think this is an issue with stunnel

sudhanshugaur4 · 2020-07-10T14:28:41Z

I am also facing same error

{ Error: Redis connection to xx.xx.x.x:6379 failed - read ECONNRESET" 

at TCP.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read'

I am also using error handler but the thing is why is this happening, it can impact my query calls and delay my requests.

@ayaz Can you please open this issue because this issue is not completely resolved people only have handled error response but the problem is still unresolved.

rujorgensen · 2020-08-07T13:03:25Z

Bumping: Same issue here:

error: Error: Redis connection to xx.yy.zz.aa:6379 failed - read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:205:27) { errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' },

robertop87 · 2020-08-13T16:15:26Z

I see the same in issue with:

NodeJS v12.16.3
NestJS framework
IOredis: 4.2
nestjs-redis: 1.2.8

Redis is provided by using Google Cloud MemoryStore.

The logs shows:

[ioredis] Unhandled error event: Error: read ECONNRESET

This is happening around one time each 24 hours without any specific reason or same steps, it just happen

stdihdev · 2020-08-24T13:14:25Z

@robertop87
I see the same issue. Redis is provided by using Google Cloud MemoryStore. Did you resolve this issue?

hkd987 · 2020-08-25T16:29:05Z

Also seeing very similar issue with Cloud run / Cloud Functions when connecting to Cloud Memory Store. It possibly could be happening when the underlying resources are stopped too soon randomly by Google.

This issue just started about a week ago for us, with no changes to how we are handling the Redis connection. We are also using ioredis to handle the connection and running node 12.

robertop87 · 2020-08-29T18:29:40Z

IORedis has as default strategy reconnect on error, but some reason it's not happening.

What I did is add the next in configuration:

reconnectOnError: (error): boolean => {
  console.log('Always reconnect on error', error)
  return true
}

The ECONNRESET error is still present, but now my connection is retried and my Google Instance is not being stuck anymore.

CC: @hkd987

Read more details about reconnectOnError here: https://github.com/luin/ioredis

hkd987 · 2020-09-02T16:37:30Z

@robertop87 Made a new deployment with the specific configuration to always reconnectOnError set to return true. This seems like an issue that needs to be reopened or a new ticket made, since this is happening on the latest release.

BridgeAR closed this as completed Sep 23, 2015

BridgeAR added the Question label Sep 23, 2015

youngboy mentioned this issue Dec 14, 2016

Improvement: Redis connection die causing node crash davidyaha/graphql-redis-subscriptions#22

Closed

MTKnife mentioned this issue Nov 27, 2017

wp2mongo hangs during article insertion. spencermountain/dumpster-dive#21

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring. #713

Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring. #713

ayazpasha2434 commented Feb 9, 2015

brycebaril commented Feb 9, 2015

wraithan commented Feb 10, 2015

ayazpasha2434 commented Feb 10, 2015

sarathms commented Feb 12, 2015

ayazpasha2434 commented Feb 13, 2015

Waterloo commented Mar 8, 2015

Waterloo commented Mar 8, 2015

blainsmith commented Aug 15, 2015

BridgeAR commented Sep 23, 2015

biggu0 commented Apr 7, 2016

BridgeAR commented Apr 7, 2016

akhare22sandeep commented Aug 2, 2016

saranyapm commented Jan 5, 2017

sw-tt-tarunvarma commented Jan 31, 2017 •

edited

Loading

riginoommen commented May 31, 2017

hametGh commented Oct 4, 2018 •

edited

Loading

stormit-vn commented Oct 22, 2018

mallikharjuna-ibexlabs commented Apr 2, 2019

emps commented May 23, 2019

sudhanshugaur4 commented Jul 10, 2020 •

edited

Loading

rujorgensen commented Aug 7, 2020

robertop87 commented Aug 13, 2020 •

edited

Loading

stdihdev commented Aug 24, 2020 •

edited

Loading

hkd987 commented Aug 25, 2020 •

edited

Loading

robertop87 commented Aug 29, 2020 •

edited

Loading

hkd987 commented Sep 2, 2020

Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring. #713

Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring. #713

Comments

ayazpasha2434 commented Feb 9, 2015

brycebaril commented Feb 9, 2015

wraithan commented Feb 10, 2015

ayazpasha2434 commented Feb 10, 2015

sarathms commented Feb 12, 2015

ayazpasha2434 commented Feb 13, 2015

Waterloo commented Mar 8, 2015

Waterloo commented Mar 8, 2015

blainsmith commented Aug 15, 2015

BridgeAR commented Sep 23, 2015

biggu0 commented Apr 7, 2016

BridgeAR commented Apr 7, 2016

akhare22sandeep commented Aug 2, 2016

saranyapm commented Jan 5, 2017

sw-tt-tarunvarma commented Jan 31, 2017 • edited Loading

riginoommen commented May 31, 2017

hametGh commented Oct 4, 2018 • edited Loading

stormit-vn commented Oct 22, 2018

mallikharjuna-ibexlabs commented Apr 2, 2019

emps commented May 23, 2019

sudhanshugaur4 commented Jul 10, 2020 • edited Loading

rujorgensen commented Aug 7, 2020

robertop87 commented Aug 13, 2020 • edited Loading

stdihdev commented Aug 24, 2020 • edited Loading

hkd987 commented Aug 25, 2020 • edited Loading

robertop87 commented Aug 29, 2020 • edited Loading

hkd987 commented Sep 2, 2020

sw-tt-tarunvarma commented Jan 31, 2017 •

edited

Loading

hametGh commented Oct 4, 2018 •

edited

Loading

sudhanshugaur4 commented Jul 10, 2020 •

edited

Loading

robertop87 commented Aug 13, 2020 •

edited

Loading

stdihdev commented Aug 24, 2020 •

edited

Loading

hkd987 commented Aug 25, 2020 •

edited

Loading

robertop87 commented Aug 29, 2020 •

edited

Loading