Redis 6 Sync Issue between master and slave

[amarsharma222]

Hi Redis Team,

We have a set up of a Master and 2 slave . Sentinel is used to monitor the cluster and take care of the automatic failover.

One of the slave which was part of the cluster had restart.
After it joined the cluster back, the read request to the slave started failing with this slave. However with the other slave the read was working fine.

We suspect that the issue we are encountering could potentially be related to the replication between the master and slave instances, particularly after the slave instance underwent a restart.

Sharing a recurring log message that we have observed multiple times:

1:S 09 May 2023 17:38:19.600 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master

Is there any known issue related to the observations in Redis 6?

Looking forward to your help in resolving the replication issue.

Regards
Amar Sharma

[amarsharma222]

@chenyang8094 and @oranagra - You have helped me in the past with an AOF file issue.Need your suggestions and advice here ,please let me know how we proceed ahead or troubleshoot with the above issue.

[oranagra]

when you say master with two slaves, are the slaves connected directly to the master, or is it a chain?
this error is returned when the slave attempts to sync with its master which is itself a slave but is not currently connected to its master.
@moticless are you aware of any recent sentinel fix that can explain that?

[moticless]

There is an attempt to promote another replica to become a master. Please verify that the log after +failover-end is +switch-master, which indicates that switch over ended successfully. Please see also if the problem happen long before, shortly before, shortly after or long after this switch over attempt.

[moticless]

To be clear, sentinel doesn't support chained replicas.

It is designed to work with a traditional master-slave replication model where multiple slave nodes replicate from a single master node. The Sentinel monitors the health of the master and slaves, and in the event of a failure, it promotes one of the slaves to act as the new master and send to all slaves slaveof <address of new master>.

[Dinesh558]

We have pods deployed as below

NAME                                                     READY   STATUS    RESTARTS   AGE    IP  
---------------------------------------------------------------------------------------------------------
redis-service-master-0                                   3/3     Running   0          14d    10.2.1.73 
redis-service-slave-0                                    3/3     Running   0          14d    10.2.0.57
redis-service-slave-1                                    3/3     Running   0          12d    10.2.2.111 

Current redis setting on each pod

master-0

role:slave
master_host:10.2.0.57
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:256821493604
slave_priority:100
slave_read_only:1
connected_slaves:1
slave0:ip=10.2.2.111,port=6379,state=online,offset=256820864962,lag=1
master_replid:e706925a7bdf536c237861a24aea528de1eadecc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:256821493604
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:256820445029
repl_backlog_histlen:1048576

slave-0

role:master
connected_slaves:1
slave0:ip=10.2.1.73,port=6379,state=online,offset=257032849903,lag=0
master_replid:e706925a7bdf536c237861a24aea528de1eadecc
master_replid2:5b60d56d8934efa3d2a60cbd1baec61c90267865
master_repl_offset:257032939271
second_repl_offset:568833705
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:257031890696
repl_backlog_histlen:1048576

slave-1

role:slave
master_host:redis-service-master-0.redis-service-headless.ns.svc.cluster.local
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:256978543265
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:e706925a7bdf536c237861a24aea528de1eadecc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:256978543265
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:256977494690
repl_backlog_histlen:1048576

It looks like, slave-1 (connected to master pod) -> master-0 (is slave and connected to slave-0) -> slave-0 (is acting as master)

[amarsharma222]

@moticless - Please let us know if the above information is helpful.

[amarsharma222]

@moticless Surprisingly there is no failover message before we performed the rollout restart. The message "Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master" has multiple occurence , and this are the logs from the slave-1 pod

[moticless]

@oranagra from what i understand from the description there was no failover attempt by sentinel to select a different master. @amarsharma222 please verify it by examining sentinel logs. As long as this is the case, then it is hard to me to see how sentinel might affect.

[moticless]

Please provide initial configuration files of replicas and sentinel before running.

[Dinesh558]

redis-default.txt
sentinel.txt

We are using Redis on kubernetes got this configuration from the base docker images

NAME READY AGE CONTAINERS IMAGES
redis-service-slave 2/2 170d redis,sentinel,metrics docker.io/bitnami/redis:6.0.5-debian-10-r10,docker.io/bitnami/redis-sentinel:6.0.5-debian-10-r9,docker.io/bitnami/redis-exporter:1.8.0-debian-10-r1

[Dinesh558]

@moticless How do we confirm or check the status of replication between pods.

[amarsharma222]

@oranagra @moticless - Your guidance in exploring potential directions that may have escaped our attention would be greatly appreciated.

[oranagra]

I don't know how you got to this configuration of using a chain of slaves, and why the middle one isn't connected to its master. Don't see how I can help.
You need to inspect your configuration and network setup.