Improve feedback when a partition is starting

[npepinpe]

Description

If a follower is far behind a leader, starting the Raft server may take a long time. Right now, the startup will simply say "Raft server is waiting to be READY", and then no other feedback is provided.

We should provide more feedback here to let users know something is happening, or at least point them where they can find out what is happening (e.g. the metrics dashboard).

Solution to be discussed in a kickoff.

[menski]

Next step: discuss possible solution, afterwards reevaluate the priority and effort

[deepthidevaki]

I tried the following. Until the server is ready, every 30 seconds it logs the current commit index. Following is an example:

2022-10-12 17:43:02.248 CEST
zeebe
RaftServer{raft-partition-partition-3} - Setting firstCommitIndex to 5112060. RaftServer is ready only after it has committed events upto this index
2022-10-12 17:43:02.250 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 4897865. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:43:32.250 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 4923669. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:44:02.254 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 4949424. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:44:32.255 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 4977335. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:45:02.288 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 5005669. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:45:32.291 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 5034895. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:46:02.292 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 5065788. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:46:32.292 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 5096333. RaftServer is ready only after it has committed events up to index 5112060
2022-10-12 17:46:47.973 CEST
zeebe
RaftServer{raft-partition-partition-3} - Commit index is 5112062. RaftServer is ready
2022-10-12 17:46:47.974 CEST
zeebe
RaftPartitionServer{raft-partition-partition-3} - Successfully started server for partition PartitionId{id=3, group=raft-partition} in 228674ms

It is logged only when commit index is update. If no new events are committed, then nothing will be logged. In that case, we can also manually increase the log level to debug and see why is stuck.

Is this enough? Do you have other ideas?

[npepinpe]

No, I think it's a good idea :) I can also imagine making use of this technique in other places 👍 (either throttling by time or number of calls 🤷)