Election scheduler should not reset before publication complete

[DaveCTurner]

Today we close the election scheduler when the coordinator leaves mode CANDIDATE, before even starting the publication that establishes the election winner as the cluster master. If this publication subsequently fails then we start a new election scheduler with the original, short, timeout, and do not back off. With very high numbers of master-eligible nodes this can lead to constant election clashes that never resolve. We must count such failed publications as failed election attempts for election scheduling and backoff purposes.

More precisely, I think a node in mode LEADER should not count an election as truly successful until we've received join votes from all nodes in the cluster (see org.elasticsearch.cluster.coordination.CoordinationState#containsJoinVoteFor) at the end of a fully-acked publication. That should be equivalent to currentTerm() == maxTermSeen at the end of a fully-acked publication, since we increase maxTermSeen on a missing join vote.

I'm not sure how a node in mode FOLLOWER should detect that the cluster is completely stable. The master could broadcast another message when it decides things are stable perhaps? Or maybe it would be good enough to base it on elapsed time (i.e. if we've been FOLLOWER in the same term for 60s)?

---

Workaround

The simplest workaround is not to have so many master-eligible nodes. See these docs for more information:

However, it is good practice to limit the number of master-eligible nodes in the cluster to three. Master nodes do not scale like other node types since the cluster always elects just one of them as the master of the cluster. If there are too many master-eligible nodes then master elections may take a longer time to complete.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[weizijun]

@weizijun that should not be necessary if you have 3 master nodes unless you have a particularly unusual configuration, and I do not want other readers to adjust this expert setting without extremely careful consideration, so I hope you don't mind that I am going to hide your comment.

yeah, It's a expert setting. so I hope this problem can be solved inside the engine without manual parameter optimization.