HDDS-7326. Intermittent timeout in TestECContainerRecovery.testContainerRecoveryOverReplicationProcessing

[captainzmc]

What changes were proposed in this pull request?

Sometimes TestECContainerRecovery.testContainerRecoveryOverReplicationProcessing will timeout, resulting in CI failure. We need to increase the inspection time to avoid this problem.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7326

How was this patch tested?

fix ut.

[adoroszlai]

Thank you @captainzmc for working on this.

How was this patch tested?

fix ut.

Sorry, but for intermittent tests a single run is not enough to ensure intermittency is gone. Can you please trigger repeated runs? @kaijchen can provide example of such run if you need help.

(Also, please search Jira for existing issue before filing one.)

[adoroszlai]

CC @swamirishi for review

[kaijchen]

Example of repeating test in CI: https://github.com/kaijchen/ozone/blob/repeat/.github/workflows/post-commit.yml#L333

[captainzmc]

Sure, Let's me trigger 10x10 iterations to test this.

[swamirishi]

This might not resolve the intermittency completely. As far as I understand I see a lot of threads in waiting state. The particular test is timing out after 100000 millis. This particular change would reduce the frequency of the test thread which would give other threads more time to perform op. Probably we should look into increasing the priority of the background threads we expect to run instead to make the test more robust. In this case which would be overReplicationHandler threads.
@adoroszlai @kaijchen What do you think?

[swamirishi]

BTW, after looking at the test case I found that issue is not the timeout but the Replication Manager Interval which is driven by this config hdds.scm.replication.thread.interval which defaults to 300s. I have fixed this issue as part of this PR. @captainzmc @adoroszlai @kaijchen Can you confirm if this is root cause for the failure.
#3939

[adoroszlai]

@swamirishi hdds.scm.replication.thread.interval setting sounds plausible as root cause. I suggest extracting that change to a separate commit, running that single test case or class repeatedly in your fork (see example). If it fixes the issue, then please create a separate PR only for the fix.

[captainzmc]

Thanks to @swamirishi @adoroszlai for the reply. Yes, I go through 10x10 iterations and find that increasing time can only reduce the probability of problems and cannot avoid error reporting. If #3939 could solves this problem, I'll turn off this PR.

[kaijchen]

Hi @captainzmc, I have just merged #3941.

[captainzmc]

Thanks @swamirishi @kaijchen @adoroszlai , Let's close this one.