OCI metrics intermittent failure(s)

[tomas-langer]

Error:  Failures: 
Error:    OciMetricsSupportTest.testEndpoint:196 
Expected: is "https://telemetry.dummyendpoint.com/"
     but: was "https://telemetry-ingestion.dummyendpoint.com/"
Error:    OciMetricsSupportTest.testMetricScope:320->validateMetricCount:423->validateMetricCount:439 
Expected: is <1>
     but: was <6>

I have seen the second issue at least 3 times

[klustria]

Been running from 100 to 200 continuous retries of the OciMetricsSupportTest like below:

 for i in {1..200}; do mvn clean test -Dtest=OciMetricsSupportTest; done 

and so far have not reproduced this issue. Will continue to attempt to reproduce this.

[klustria]

The use of @RepeatedTest(4000) on testEndpoint helped reproduce the problem and I think I have now an understanding of the behavior:

1. The failure in OciMetricsSupportTest.testEndpoint is a race condition wherein the restore of the endpoint in the OciMetricsSupport has not been executed yet before the test assert call is made to validate it. I think that the test should wait for a certain period of time (maybe a few seconds even though it probably just missed it by a few milliseconds) to validate that the endPoint has been restored.
2. The failure in OciMetricsSupportTest.testMetricScope is a chain reaction effect of the above problem. Because it failed, the webServer was not stopped and so the ociMetricsSupport executor thread will continue to be running and at a certain interval, will be retriggered, and will call countDownLatch1.countDown() which will affect one of the validations in OciMetricsSupportTest.testMetricScope. Because countDownLatch1 is a global static variable, the one used by the failed test will be reused by the new test causing this failure. We can ignore this and fix the above issue and this will not be a problem anymore or make countDownLatch1 as test method local variable.

[klustria]

Problem can be reproduced consistently by using these steps:

1. Replace @Test with @RepeatedTest(4000) in OciMetricsSupportTest.testEndpoint
2. Run a mvn test executing OciMetricsSupportTest.testEndpoint and OciMetricsSupportTest.testMetricScope 5 times or until it fails using this one liner bash command:

   for i in {1..5}; do mvn clean test -Dtest=OciMetricsSupportTest#testEndpoint+testMetricScope || break && echo "Finished test #$i"; done;

[klustria]

PR #6151 has been created as potential fix for this issue.

[klustria]

PR #6151 merged, hence closing.