raft: implement leader steps down

[xiang90]

From CONSENSUS: BRIDGING THEORY AND PRACTICE by diego.

A server might be in the leader state, but if it isn’t the current leader, it could be needlessly delaying client requests. For example, suppose a leader is partitioned from the rest of the cluster, but it can still communicate with a particular client. Without additional mechanism, it could delay a request from that client forever, being unable to replicate a log entry to any other servers. Meanwhile, there might be another leader of a newer term that is able to communicate with a majority of the cluster and would be able to commit the client’s request.

Thus, a leader in Raft steps down if an election timeout elapses without a successful round of heartbeats to a majority of its cluster; this allows clients to retry their requests with another server.

This is important to etcd since the pervious leader needs to cancel all connected watchers to prevent forever waiting.

/cc @bdarnell @gyuho

[gyuho]

Just to clarify, is this issue about improving the watch(er) implementation? Or do we need change on client implementation as well? Thanks,

[xiang90]

Just to clarify, is this issue about improving the watch(er) implementation?

First, we need to improve raft. So that the leader can actually step down itself. Second, we need to improve watch implementation: if a server is partitioned, it should cancel all its watchers to avoid forever watching. Third, the new lease feature also needs leader stepping down.

Or do we need change on client implementation as well? Thanks,

I do not have plan for client side implementation right now.

[gyuho]

Makes sense. Thanks for clarification!