Node stays leader after role changed to RAFT_SPARE

[MathieuBordere]

A node stays leader of a cluster after being demoted to RAFT_SPARE. This results in unavailability of a dqlite cluster because a node will only report who the leader if it is a voter in the current configuration, see https://github.com/canonical/dqlite/blob/9946636b7b6e3d41ad463c6ca11443c6e026ea49/src/gateway.c#L151

The other nodes will still think the spare node is the leader and will refer the client to the spare node, who in turn will report that it doesn't know the leader.

There's a test in raft with RAFT_STANDBY role instead of RAFT_SPARE https://github.com/canonical/raft/blame/master/test/integration/test_assign.c#L435, so this behavior looks intentional.

This looks to be the cause of canonical/dqlite#323.

@freeekanayaka I think the leader should step down once it notices that it's no longer a voter in the newly committed configuration, what do you think?

[freeekanayaka]

Yes, it should step down indeed. It's pretty much the same as if it was removed from the configuration altogether, as per section 4.2.2 of the dissertation ("a leader that is removed from the configuration steps down once the Cnew entry is committed").

[tomponline]

thanks @freeekanayaka