Flaky-test: SaslAuthenticateTest.testSaslOnlyAuthFirstStage

[storm-08]

Search before reporting

• I searched in the issues and found nothing similar.

Example failure

https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e

Exception stacktrace

Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.

java.lang.AssertionError:
Expected :10
Actual   :7
<Click to see difference>


	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.failNotEquals(Assert.java:1577)
	at org.testng.Assert.assertEqualsImpl(Assert.java:149)
	at org.testng.Assert.assertEquals(Assert.java:131)
	at org.testng.Assert.assertEquals(Assert.java:1418)
	at org.testng.Assert.assertEquals(Assert.java:1382)
	at org.testng.Assert.assertEquals(Assert.java:1428)
	at org.apache.pulsar.broker.authentication.SaslAuthenticateTest.testSaslOnlyAuthFirstStage(SaslAuthenticateTest.java:332)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)

Are you willing to submit a PR?

• I'm willing to submit a PR!

Description

testSaslOnlyAuthFirstStage can intermittently fail due to a strict cache size assertion affected by entry expiry timing.

Expected Behaviour

The test should reliably verify that some entries exist in the cache without assuming exact timing.

Actual Behaviour

Test sometimes fails when entries from the first loop expire before the size assertion, leading to less than 10 in the cache.

Steps to reproduce

Simply add Thread.sleep(5) before the end of first for loop and then run the test case.

Notes

• The expiry time (inflightSaslContextExpiryMs) is set to 50ms.
• The first loop can take >50ms, causing early entries to be evicted before the assertion.
• The strict assertEquals(cache.size(), 10) made the test flaky.

Please see this for the changes.

Link to detailed logs and trace of error - https://gist.github.com/storm-08/61ff3104a1f0492cfa98c600708faa3e

Please note I have added Thread.sleep(5) in my test case so line number may offset by 1 in trace.