-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
Description
When using Redis Sentinel, after Redis Sentinel went down, Redis client doesn't reconnect after it come back online.
This is a regression we noticed when upgrading from @redis/client 5.8.3 to 5.9.0.
Minimal reproducible script: simply call client.get() every 3 seconds and print out the result
const { createSentinel } = require('@redis/client');
const client = createSentinel({
name: 'mymaster',
sentinelRootNodes: [{ host: '192.168.11.123', port: 26379 }],
RESP: 3,
});
client
.on('connect', () => console.info('Redis connected'))
.on('end', () => {
console.info('Redis disconnected, reconnecting');
client.connect();
})
.on('error', (err) => console.error('Redis error: ' + err))
.connect();
setInterval(async () => {
const timestamp = new Date().toISOString();
try {
const results = await Promise.race([
client.get('test-key'),
new Promise((_, reject) => setTimeout(() => reject(new Error('1s Timeout')), 1000)),
]);
console.log(`${timestamp} - Redis GET success: ${results}`);
} catch (err) {
console.log(`${timestamp} - Redis GET fail: ${err.message}`);
}
}, 3000);Example log of running the script -> shutdown Redis cluster (Sentinel, Master, Slave) -> restart the cluster
(or simply disconnecting - wait - reconnect ethernet cable)
v5.9.0
$ npm i @redis/client@5.9.0
$ node redis-disconnect.js
2025-11-05T14:01:22.293Z - Redis GET success: null
2025-11-05T14:01:25.295Z - Redis GET success: null
2025-11-05T14:01:28.301Z - Redis GET fail: 1s Timeout
2025-11-05T14:01:31.304Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
2025-11-05T14:01:34.311Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
Error: None of the sentinels are available
at RedisSentinelInternal.observe (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:759:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async #connect (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:522:51)
2025-11-05T14:01:37.318Z - Redis GET fail: 1s Timeout
2025-11-05T14:01:40.325Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
2025-11-05T14:01:43.327Z - Redis GET fail: 1s Timeout
2025-11-05T14:01:46.329Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
Error: None of the sentinels are available
at RedisSentinelInternal.observe (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:759:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async #connect (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:522:51)
2025-11-05T14:01:49.330Z - Redis GET fail: 1s Timeout
and it never recover, even after the Redis cluster is back online. We need to manually restart the app.
v5.8.3, it automatically reconnects just fine:
$ npm i @redis/client@5.8.3
$ node redis-disconnect.js
2025-11-05T14:06:12.746Z - Redis GET success: null
2025-11-05T14:06:15.746Z - Redis GET success: null
Redis error: obseve client error: Error: connect ECONNREFUSED 192.168.11.123:26379
Redis error: Error: connect ECONNREFUSED 192.168.11.123:26379
2025-11-05T14:06:21.751Z - Redis GET fail: 1s Timeout
2025-11-05T14:06:24.751Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
2025-11-05T14:06:27.754Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
Error: None of the sentinels are available
at RedisSentinelInternal.observe (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:752:15)
at processTicksAndRejections (node:internal/process/task_queues:105:5)
at runNextTicks (node:internal/process/task_queues:69:3)
at process.processTimers (node:internal/timers:520:9)
at async #connect (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:522:51)
2025-11-05T14:06:30.758Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: connect ECONNREFUSED 192.168.11.123:26379
Redis error: Error: connect ECONNREFUSED 192.168.11.123:26379
2025-11-05T14:06:33.761Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
2025-11-05T14:06:36.764Z - Redis GET fail: 1s Timeout
2025-11-05T14:06:39.765Z - Redis GET fail: 1s Timeout
Redis error: obseve client error: Error: Connection timeout
Redis error: Error: Connection timeout
Error: None of the sentinels are available
at RedisSentinelInternal.observe (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:752:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async #connect (/.../node_modules/@redis/client/dist/lib/sentinel/index.js:522:51)
2025-11-05T14:06:42.769Z - Redis GET success: null
2025-11-05T14:06:45.775Z - Redis GET success: null
This may be related to #3108
(side note, notice how neither "Redis connected" nor "Redis disconnected" is in the log: #3012)
Adding
.on('end', () => {
console.info('Redis disconnected, reconnecting');
client.connect();
});doesn't help either as the signal isn't triggered
Node.js Version
22.21.1
Redis Server Version
8.2.3
Node Redis Version
5.9.0
Platform
Linux