Flaky-test: ProxyPrometheusMetricsTest.testMetrics

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/actions/runs/3905486374/jobs/6672708495#step:11:441

Exception stacktrace

  Error:  Tests run: 6, Failures: 1, Errors: 0, Skipped: 5, Time elapsed: 14.542 s <<< FAILURE! - in org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest
  Error:  testMetrics(org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest)  Time elapsed: 0.031 s  <<< FAILURE!
  java.lang.IllegalArgumentException: Failed to register Collector of type Counter: test_counter_total is already in use by another Collector of type Counter
  	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57)
  	at io.prometheus.client.Collector.register(Collector.java:307)
  	at io.prometheus.client.Collector.register(Collector.java:300)
  	at org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest.testMetrics(ProxyPrometheusMetricsTest.java:99)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

Reopening the issue. After applying #19217, the actual problem shows up.

  Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 15.921 s <<< FAILURE! - in org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest
  Error:  testMetrics(org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest)  Time elapsed: 0.079 s  <<< FAILURE!
  java.lang.AssertionError: expected [1] but found [0]
  	at org.testng.Assert.fail(Assert.java:110)
  	at org.testng.Assert.failNotEquals(Assert.java:1413)
  	at org.testng.Assert.assertEqualsImpl(Assert.java:149)
  	at org.testng.Assert.assertEquals(Assert.java:131)
  	at org.testng.Assert.assertEquals(Assert.java:1240)
  	at org.testng.Assert.assertEquals(Assert.java:1274)
  	at org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest.testMetrics(ProxyPrometheusMetricsTest.java:118)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)

[lhotari]

This remains flaky, even after adding Awaitility. example https://github.com/lhotari/pulsar/actions/runs/3931620216/jobs/6723297782#step:11:452

  Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 30.856 s <<< FAILURE! - in org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest
  Error:  testMetrics(org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest)  Time elapsed: 10.429 s  <<< FAILURE!
  org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest expected [1] but found [0] within 10 seconds.
  	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
  	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
  	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
  	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
  	at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:769)
  	at org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest.testMetrics(ProxyPrometheusMetricsTest.java:111)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)
  Caused by: java.lang.AssertionError: expected [1] but found [0]
  	at org.testng.Assert.fail(Assert.java:110)
  	at org.testng.Assert.failNotEquals(Assert.java:1413)
  	at org.testng.Assert.assertEqualsImpl(Assert.java:149)
  	at org.testng.Assert.assertEquals(Assert.java:131)
  	at org.testng.Assert.assertEquals(Assert.java:1240)
  	at org.testng.Assert.assertEquals(Assert.java:1274)
  	at org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest.lambda$testMetrics$1(ProxyPrometheusMetricsTest.java:116)
  	at org.awaitility.core.AssertionCondition.lambda$new$0(AssertionCondition.java:53)
  	at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:248)
  	at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:235)
  	... 4 more

[codelipenghui]

Another one: https://github.com/apache/pulsar/actions/runs/3935561394/jobs/6731481016

[lhotari]

This problem seems to be caused by #19062 .

[labuladong]

Seems it's still flaky:

https://github.com/apache/pulsar/actions/runs/3937453167/jobs/6795361656