HDDS-10582. Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI #6585
Conversation
| GenericTestUtils.waitFor(() -> TestHelper.countReplicas(containerIdR3, cluster) == 4, | ||
| 200, 30000); |
There was a problem hiding this comment.
Observed the condition was true for a very smaller span
The same problem may affect other tests (currently there is only one other usage). I think the new, reduced check interval can be set in waitForReplicaCount.
There was a problem hiding this comment.
That is why I cross checked the uses of waitForReplicaCount which I found in TestContainerReplication, And I tried to see its flakiness..
- Worked 100% success for 10x10 https://github.com/raju-balpande/apache_ozone/actions/runs/8710014704
- Worked 99.75% success for 20x20 https://github.com/raju-balpande/apache_ozone/actions/runs/8720977974
And seems no change require for this class.
Please suggest if I shall make this change in TestHelper.waitForReplicaCount instead.
There was a problem hiding this comment.
no change require for this class
If it's not getting worse due to the change, please make it in waitForReplicaCount.
There was a problem hiding this comment.
I agree to your point. Even though the TestContainerReplication is not breaking now but the following two lines are indicating the condition is true for short span which if missed will fail the test method and hence thought to extend the change for this class as well. Hence making change in TestHelper.waitForReplicaCount only.
Will be observing the result for both the flaky classes under https://github.com/raju-balpande/apache_ozone/actions/runs/8831034232
…n TestContainerReplication
…conAndAdminContainerCLI (apache#6585) (cherry picked from commit dd86223)
What changes were proposed in this pull request?
Intermittent timeout during waitForReplicaCount in TestReconAndAdminContainerCLI
I observed two wait condition having similar checks and were intermittently failing. Observed the condition was true for a very smaller span and hence increased the frequency to check. Earlier frequency was 1000ms, tested it with 500ms and 200ms. 500ms still have few failures. 200ms is working with no failure.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10582
How was this patch tested?
Test it in CI by creating flaky tests check : https://github.com/raju-balpande/apache_ozone/actions/runs/8817120192