Offline queue doesn't works when redis server restarting.

[Dr-klo]

Description

Hi. I'm trying to configure redisClient to flush the commands which was cached during redis server were offline but can't
Below is my code. I'm trying to write values in infinity loop. I'm start the code and then restart docker container with redis instance. After that If I check test:foo set I saw 1,2,3,4,26,27,28 ..... So I miss 5-25 values or similar which the code tried to write while the server were off.
How to configure offlineQueue correctly?

const redisClient = redis.createClient({
  socket:{
  host:config.redis.host, 
  port:config.redis.port,
  reconnectStrategy:  retries => Math.min(retries * 50, 5000)},
  commandsQueueMaxLength: 99999,
  disableOfflineQueue:false,
});
redisClient.on('error', error => console.log(`Redis exception on worker ${process.env.workerType} (pid  ${process.pid}): ${error}`));

async function loop(i){
  try{
    console.log('loop '+i);
    let multi = redisClient.multi();
    await multi.zAdd('test:foo', {score:i, value:i.toString()}).exec();
  }catch (e) {
    console.log('catch',e)
  }
}
(async()=>{
  await redisClient.connect()
  let i = 0;
  setInterval(()=> loop(++i), 1000);
})();

Node.js Version

v14.21.3

Redis Server Version

7.0.13

Node Redis Version

4.6.12

Platform

Windows

Logs

No response

[leibale]

the client won't retry commands after they were written to the socket even if the offline queue is on because the client cannot know if the command was executed on the server or not (and rerunning commands like INCR without some checks is not a good idea..).
you should see "catch" logs for the commands those commands, and you can execute them again on the application level.

[mgsxman]

@leibale Well, but what is the purpose of the offline queue then? I also do not quite understand that.
According to docs it should store (add to the queue) the commands while the connection is offline/down and retry/flush the queue after the connection been restored to the Redis server?

[leibale]

@mgsxman the client will write each command to the socket once. If the client wrote a command to the socket, then the socket drops, the client can't know if the command was executed on the server or not, and therefore won't retry them...
Using offline queue, when the socket drops, all the commands that weren't written to to socket yet will wait for a new socket.

[mgsxman]

@leibale Right, but according to the example mentioned above, the socket been dropped by shutting down the Redis server. After that during a certain period of time the client tried to write commands to the dropped socket and therefore were not written successfully yet because the socket already been dropped. And my assumption is that it should have been queue the remaining commands 10,11,12...25 for the new socket, ie. when the Redis server started again and retry those commands after the connection is established.
Am I understanding correctly?

But that did not happen, and all of the commands been removed and never retried. Why ?
The whole process of the offline queue is so confusing.

[leibale]

You are describing "offline queue" correctly, but missing some other things:

1. "After that during a certain period of time the client tried to write commands to the dropped socket and therefore were not written successfully" - a "successful write" is writing to the socket, there is no guarantee on the other side receiving it (let alone process and execute it) until you get a reply.
2. You are running your commands using setInterval, which means that (for example) round 2 does not wait for round 1 to finish..

Using this code:

import { createClient } from 'redis';
import { setTimeout } from 'node:timers/promises';

const client = await createClient()
  .on('error', err => console.error('Redis client error', err))
  .connect();

let i = 0;
while (true) {
  const current = i++;
  
  try {
    console.log(
      current,
      await client.set(current.toString(), '')
    );
  } catch (err) {
    console.error(current, err);
  }

  await setTimeout(500);
}

when I restart the docker while the script is running I get one of the two:

1. there is 1 missing key in redis
2. I am getting very "lucky" and all the keys are in the db (because the client "realized" the socket is down due to TCP keep-alive).