RATIS-1622. Fix high CPU load when some followers are down

[drriguz]

What changes were proposed in this pull request?

Avoid using 0 delay for slow or dead followers when new entries comes.

What is the link to the Apache JIRA

RATIS-1622: High cpu usage of LogAppenderDaemon

Please replace this section with the link to the Apache JIRA)

How was this patch tested?

Tested manually by using the following steps:

• Start a cluster of 5 nodes
• shutdown node 5 and 5
• make some change to get new log entry
• monitor the CPU usage of the leader node.

The CPU load is significantly reduced from 90% to 10%(in Docker environment, and cpus is limited to 1 for each node).

However, some unit tests fails locally. I'll open this to see if any tests will fail in CI.

[szetszwo]

@drriguz , the change looks good. There is an unused import (see the checkstyle warning) and a typo in the comment.

[szetszwo]

+1 the change looks good.

[codings-dan]

@drriguz There are many failed ci tests, see https://github.com/apache/ratis/runs/7333835180?check_suite_focus=true#step:5:1030 Can you take a look, thank you!

[drriguz]

@drriguz There are many failed ci tests, see https://github.com/apache/ratis/runs/7333835180?check_suite_focus=true#step:5:1030 Can you take a look, thank you!

Yeah, I'll try to figure it out!

[drriguz]

@szetszwo Hi, there's failed test:

Error:  testRestartFollower(org.apache.ratis.grpc.TestInstallSnapshotNotificationWithGrpc)  Time elapsed: 62.907 s  <<< ERROR!
java.lang.IllegalArgumentException: s2@group-1E4B33D1632B-SegmentedRaftLog is expected to be opened but it is CLOSED

previously the failed test is:

TestNettyDataStreamStarTopologyWithGrpcCluster>DataStreamAsyncClusterTests.testMultipleStreamsMultipleServersStepDownLeader:78->DataStreamAsyncClusterTests.runTestDataStreamStepDownLeader:82->DataStreamAsyncClusterTests.runTestDataStream:97 » TestTimedOut

But I tried run tests locally several times and none of them failed, either the tests are not stable, or this commit somehow makes it unstable. Do you think it's related to this commit?

[adoroszlai]

@drriguz only TestServerRestartWithSimulatedRpc is failing consistently:

https://github.com/apache/ratis/runs/7350397280?check_suite_focus=true#step:5:525
https://github.com/apache/ratis/runs/7350824719?check_suite_focus=true#step:5:525
https://github.com/apache/ratis/runs/7350984684?check_suite_focus=true#step:5:525
https://github.com/apache/ratis/runs/7351204878?check_suite_focus=true#step:5:525

[drriguz]

@drriguz only TestServerRestartWithSimulatedRpc is failing consistently:

https://github.com/apache/ratis/runs/7350397280?check_suite_focus=true#step:5:525 https://github.com/apache/ratis/runs/7350824719?check_suite_focus=true#step:5:525 https://github.com/apache/ratis/runs/7350984684?check_suite_focus=true#step:5:525 https://github.com/apache/ratis/runs/7351204878?check_suite_focus=true#step:5:525

Yes, then it must be related. I'll dig into it.