Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
LogMatch invariant violated #58
Through fuzz testing I found a test case that causes Raft's "Log Matching" invariant to be violated. "Log Matching" is defined as:
One issue seems to be the following: Upon transitioning from Candidate to Leader, the Candidate sends an
The fix should be simple: initialize the
In more detail:
I believe that #37 describes another issue here: prevLogIndex should be -1 for an empty log, not 0 [or, alternatively, the log should be 1-indexed instead of 0-indexed]. If followers understood that prevLogIndex should be 0 for a log containing one entry, they would have failed the consistency check for the AppendEntries command, and properly rejected it.
Here are the reproducing steps for this execution:
Here are my annotated notes on each step of the execution.
For what it's worth, this snapshot of the akka-raft code differs slightly from the master branch. In particular, it includes (hacky) fixes for all of the known leader election bugs, and a few printlns within raft.
After thinking about this a bit more:
Delayed nextIndex initialization is not the main issue here. That causes a separate issue (IndexOutOfBounds exception at the leader because its nextIndex gets into a non-sensical state).
The main issue here is that the initial value of prevLogIndex is ambiguous: followers can't distinguish between an AppendEntries for an empty log (prevLogIndex == 0), an AppendEntries for the leader's 1st command (prevLogIndex == 0), and an AppendEntries for the leader's 2nd command (prevLogIndex == 1 - 1 == 0). The last two cases need to be distinguishable. Otherwise followers won't be able to reject inconsistent logs.
For what it's worth, I have a (non-pull-request-worthy) fix for this issue here:
Tested by fuzz testing -- I verified that under non-failing scenarios, this behaves correctly at each step of the execution when replicating two client commands.