Total data loss on failover

[aphyr]

With Redis f88f866 and Redisraft 1b3fbf6, every failover results in the complete loss of all data in the cluster. This can occur both in healthy clusters due to sporadic leader elections, by pausing or killing nodes, network partitions, etc. redis monitor does show leader-executed commands on followers, but I'm not clear if that's being proxied from the Raft leader, replicated but not actually applied to the local state machine, applied but then destroyed somehow, or what.

For example:

$ ssh n1 /opt/redis/redis-cli set x foo
OK
$ ssh n1 /opt/redis/redis-cli set y bar
OK
$ ssh n1 /opt/redis/redis-cli set z baz
OK
$ ssh n1 /opt/redis/redis-cli set t xyzzy

^C⏎

Here, a leader election occurred during our set of t. We retry on n2...

$ ssh n1 /opt/redis/redis-cli set t xyzzy
MOVED 192.168.122.12:6379

$ ssh n2 /opt/redis/redis-cli set t xyzzy
OK

And lo and behold, all the keys we wrote before are gone.

$ ssh n2 /opt/redis/redis-cli keys "'*'"
t

If we kill n2 as a leader, even that disappears.

$ ssh n2 killall -9 redis-server
$ ssh n3 /opt/redis/redis-cli keys "'*'"
__raft_snapshot_info__

[yossigo]

Your observation is correct, the problem in this (and the previous) case is with applying the commands to Redis. A missing re-entrancy check causes the applied commands to be re-raftized (i.e. sent again to the Raft log instead of to the FSM).

[aphyr]

Confirmed--this looks fixed as of d589127