HDDS-3227. Ensure eviction of stateMachineData from cache only when both followers catch up

[bshashikant]

What changes were proposed in this pull request?

Changing the stateMachine caching gurantees.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3227

How was this patch tested?

Existing UTs

[lokeshj1703]

@bshashikant Thanks for working on this! I think we should make this change configurable along with the follower gap threshold in Ratis.

[lokeshj1703]

@bshashikant Thanks for updating the PR! I have a few comments inline.

[lokeshj1703]

@bshashikant Thanks for updating the PR! The changes look good to me. Only a minor comment. +1 o.w.

[lokeshj1703]

Rebase on master might help with failing CI.

[bshashikant]

The test failures are not related to the patch.

[adoroszlai]

It seems TestRandomKeyGenerator is timing out consistently (3/3 runs now). Problem happens with the 2GB key test case (Key size: 2147483657 bytes). Can be reproduced locally, too.

[lokeshj1703]

@bshashikant Thanks for the contribution! @bharatviswa504 @adoroszlai Thanks for the review! I have committed the PR to master branch.

[adoroszlai]

@lokeshj1703 @bshashikant

While TestRandomKeyGenerator did not time out in the last PR run before the merge, I think it still shows some problem. It took more than twice longer than previously:

Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 443.256 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator

vs.

Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.665 s - in org.apache.hadoop.ozone.freon.TestRandomKeyGenerator

And it has timed out on master after the PR was merged:
https://github.com/apache/ozone/runs/4440507728?check_suite_focus=true#step:4:3010

Running it locally, I see appendEntries Timeout in the log.

[adoroszlai]

OK, I think I found the problem:
https://issues.apache.org/jira/browse/HDDS-6071

[bshashikant]

OK, I think I found the problem: https://issues.apache.org/jira/browse/HDDS-6071

Thanks @adoroszlai for root causing the issue.