Avoid expiration and eviction during data syncing #13077
Conversation
There was a problem hiding this comment.
i didn't take a deep look at this PR yet, but we do already have this in our fork, so maybe it's easier and safer to PR that than review another implementation.
@YaacovHazan FYI.
src/replication.c
Outdated
| } else if (!strcasecmp(c->argv[j]->ptr,"pseudo-replica")) { | ||
| /* REPLCONF PSEUDO-REPLICA is used to set this replica | ||
| * into 'pseudo-replica' mode to avoid eviction and expiration. | ||
| * This is used for sync tools which eviction and expiration may | ||
| * cause the data corruption. */ | ||
| long pseudo_replica = 0; |
There was a problem hiding this comment.
REPLCONF is used to set the client state of the calling client, not the server state.
this means that pseudo-replica should be a config (config.c).
And REPLCONF should have a pseudo-master to set the current connection behave like a master.
if we tie them together, it may mean we need to reset the server state when that client drops, which not necessarily desired since it may soon re-connect and we still don't want eviction to kick in.
There was a problem hiding this comment.
Thanks for your review. I have separated these two logics.
@oranagra Very glad that you have a plan to PR that :). Before that, I'm willing to continue this work. |
|
|
When we sync data from the source Redis to the destination Redis using some sync tools like redis-shake, the destination Redis think it's a master and can perform expiration and eviction, which may cause data corruption like what @oranagra said in #9760 (reply in thread).
In standalone mode, we can use writable replica to simplify the sync process. However, in cluster mode, we still need a sync tool to help us transfer the source data to the destination. The sync tool usually work as a normal client so the destination works as a master which keep expiration and eviction.
In this PR, we add a new mode for master named 'pseudo-replica'. In this mode, server stop expiration and eviction just like a replica. Notice that this mode exists only in sync state to avoid data inconsistency caused by expiration and eviction.