RATIS-1490. Follower should not change to candidate if already stopped

[adoroszlai]

What changes were proposed in this pull request?

RaftServerImpl creates a new FollowerState for each vote request received. The old FollowerState may trigger change to candidate role even after it was shutdown by the server. Both RaftServerImpl#requestVote and the change to candidate by FollowerState requires lock on server (RaftServerImpl object). The order of events which triggers the problem:

• RaftServerImpl executing requestVote with lock on itself
• FollowerState checks conditions, then waits for lock on server
• RaftServerImpl shuts down old FollowerState, starts new one
• RaftServerImpl sets last RPC time on new FollowerState
• RaftServerImpl releases lock
• old FollowerState enters block with lock on server, triggers change to candidate

Let FollowerState check if it is still running after entering the synchronized block.

https://issues.apache.org/jira/browse/RATIS-1490

How was this patch tested?

Tweaked FollowerState, RaftServerImpl and LeaderElectionTests with code injection. It required too much test-specific code, even in synchronized blocks, so I think it should not be part of this patch. Ideas for unit tests welcome.

Regular CI:
https://github.com/adoroszlai/incubator-ratis/runs/4845475132

[codings-dan]

I think this change is good, but it doesn't seem to be very easy to test it. Is it possible to consider adding a way to start Application in software like Intellij IDEA and start several ratis servers locally, which can easily add breakpoints and reproduce the problem, which is helpful for debugging and testing.

[szetszwo]

+1 the change looks good.

[adoroszlai]

Thanks @codings-dan, @szetszwo for the review, @szetszwo for merging this.