transfer leader may not be safe for lease read

[BusyJay]

if messages are out of order and MsgTimeoutNow arrives after an election timeout, a new leader may be elected probably as MsgTimeoutNow will trigger election despite of the lease. So even if there is no clock drift, leader's lease can still be stale.

The problem is that when aborting transferring leader, leader just resets its internal field, which is not enough, because MsgTimeoutNow may still being transferring. So leader has to make sure the MsgTimeoutNow message becomes stale. It can either start a new election or add last index and term to MsgTimeoutNow and broadcast an empty entry. Former will make the election triggered by MsgTimeoutNow can't succeed, latter will either make the election can't succeed or MsgTimeoutNow be dropped.

/cc @siddontang

[BusyJay]

/cc @tiancaiamao

[jayzhan211]

add last index and term to MsgTimeoutNow and broadcast an empty entry

1. what is the purpose of broadcast an empty entry
2. if leader election timeout does not change index or term, but only set transferee to None, I dont think leader_transferee can check term or index in order to drop MsgTimeoutNow

raft-rs/src/raft.rs

Lines 838 to 848 in f145d5a

	if self.election_elapsed >= self.election_timeout {
	self.election_elapsed = 0;
	if self.check_quorum {
	let m = new_message(INVALID_ID, MessageType::MsgCheckQuorum, Some(self.id));
	has_ready = true;
	let _ = self.step(m);
	}
	if self.state == StateRole::Leader && self.lead_transferee.is_some() {
	self.abort_leader_transfer()
	}
	}

[BusyJay]

When last index and term are recorded in MsgTimeoutNow, transferee can check if it still matches its log. If not, it can be dropped directly. Broadcast an empty entry when about to abort transferring is to make either quorum have more logs than transferee or transferee's log more up to date than the MsgTimeoutNow.

[jayzhan211]

how to deal with the situation that,
if empty entry that broadcast by leader is received after MsgRequestVote from transferee, then the follower will send MsgRequestVoteResponse to transferee, and transferee will become leader, which is not expected, (after that follower receive broadcast message and update the log but it is too late)

I have an answer,

leader election timeout will send empty entry MsgAppend and receive MsgAppendResponse from followers,
transferee will receive MsgTimeoutNow and send MsgRequestVote and receive MsgRequestVoteResponse,

leader will abort transferee only after receiving MsgAppendResponse, it means the followers has up -to-date logs, otherwise transfer leadership is considered success and transferee will get the follower's MsgRequestVoteResponse