raft: pre-candidate can spuriously win election from vote in prior term

[nvanbenschoten]

From #123189 (comment), see the PreVote bug:

I believe that there is a bug in the existing PreVote algorithm in etcd/raft (and our new fork) that can allow a pre-vote candidate to win an pre-vote election that it otherwise shouldn't. Specifically, pre-vote elections are called with the term following the current term, which is carried by MsgPreVote messages. This term is also carried in non-rejected MsgPreVoteResp messages. Pre-vote candidates know to handle these non-rejected MsgPreVoteResp messages with a future term. However, they have no special handling of non-rejected MsgPreVoteResp messages with their current term. These messages correspond to a pre-vote election held at the previous term, but it looks like they can be used to win a pre-election at the current term.

For example, I think the following order of operations is allowed:

1. n3 is the leader at term 10
2. n1 calls a pre-vote election at term 10, sends MsgPreVote{Term: 11} to n2 and n3
3. n2 responds with a successful MsgPreVoteResp{Term: 11, Reject: false}, but the message is delayed
4. n1 never wins the pre-election
5. eventually the term advances to term 11 and n2 takes over as leader
6. n1 calls a pre-vote election at term 11, sends MsgPreVote{Term: 12} to n2 and n3
7. n2 and n3 both reject these, however, before they can respond...
8. the original message from n3 is delivered, n1 thinks it won the pre-vote election
9. n1 calls a disruptive election for term 12

The consequences for this bug are not overly severe, because a pre-vote election is a best-effort attempt to prevent a disruptive leader election. Bugs in pre-vote do not compromise safety (i.e. leader exclusivity). Still, we should fix this.

Jira issue: CRDB-38276

[blathers-crl]

cc @cockroachdb/replication