about fail node resume

[xichehe]

I have two rack(rack1 rack2) and each have six node,each node have one redis service on backend.
my client set & read data with phpredis extension.

test case 1:
rack1 & rack2 all up, random set & get data to all node, all is ok,no problem.
test case 2:
rack1 up , rack 2 some node down, backend redis up,random set & get data all rack1 node(avoid all rack 2 node),still no problem.
but rack1 and rack2 node data is diffent.
teste case 3:
rack1 up,resume rack 2 down node,random get data to all node, some test case 2 set data is missing.

my question：

fail node resume, rack1 and rack2 node data is diffent,How can avoid to get some key is missing?

my config

node 1:

dyn_o_mite:
datacenter: default_dc
dyn_listen: 0.0.0.0:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

• 72.1.121.12:8101:rc0:default_dc:3346660200
• 72.1.121.16:8101:rc1:default_dc:1656170642
• 72.1.121.13:8101:rc0:default_dc:278415870
• 72.1.121.14:8101:rc1:default_dc:2691937952
• 72.1.121.15:8101:rc1:default_dc:2565460974
  gos_interval: '500'
  listen: 0.0.0.0:6379
  rack: rc0
  redis: 'true'
  servers:
• 127.0.0.1:6380:1
  timeout: 150000
  tokens: '400517191'

node 2:

dyn_o_mite:
datacenter: default_dc
dyn_listen: 0.0.0.0:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

• 72.1.121.11:8101:rc0:default_dc:400517191
• 72.1.121.16:8101:rc1:default_dc:1656170642
• 72.1.121.13:8101:rc0:default_dc:278415870
• 72.1.121.14:8101:rc1:default_dc:2691937952
• 72.1.121.15:8101:rc1:default_dc:2565460974
  gos_interval: '500'
  listen: 0.0.0.0:6379
  rack: rc0
  redis: 'true'
  servers:
• 127.0.0.1:6380:1
  timeout: 150000
  tokens: '3346660200'

.....

node 6

dyn_o_mite:
datacenter: default_dc
dyn_listen: 0.0.0.0:8101
dyn_port: 8101
dyn_seed_provider: simple_provider
dyn_seeds:

• 72.1.121.11:8101:rc0:default_dc:400517191
• 72.1.121.12:8101:rc0:default_dc:3346660200
• 72.1.121.16:8101:rc1:default_dc:1656170642
• 72.1.121.13:8101:rc0:default_dc:278415870
• 72.1.121.14:8101:rc1:default_dc:2691937952
  gos_interval: '500'
  listen: 0.0.0.0:6379
  rack: rc1
  redis: 'true'
  servers:
• 127.0.0.1:6380:1
  timeout: 150000
  tokens: '2565460974'

[ipapapa]

The reason that you have multi-rack deployment is for failover. Evidently, there will be cases that the two racks may be out of sync. If you do not have a lot of writes, you can stop the writes on that database, and quickly sync up the nodes using the persistence feature in Redis: redis-cli bgrewriteaof. We are also working on a scalable approach.