Flaky-test: ResendRequestTest. testExclusiveCumulativeAckedNormalTopic

[RobertIndie]

ResendRequestTest.testExclusiveCumulativeAckedNormalTopic is flaky. It fails sporadically.

example failure

Error:  testExclusiveCumulativeAckedNormalTopic(org.apache.pulsar.broker.service.ResendRequestTest)  Time elapsed: 60.166 s  <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.ResendRequestTest.testExclusiveCumulativeAckedNormalTopic() didn't finish within the time-out 60000
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371)
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605)
	at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[shibd]

I reproduced it,I'll try to fix it.

[shibd]

https://github.com/apache/pulsar/runs/5417203130?check_suite_focus=true#step:9:2179

[shibd]

PIP-84

In the current implementation, the consumer epoch and broker epoch may be inconsistent. Unless reconnected, otherwise consumers will always filter message.

When the consumer calls the consumer.redeliverUnacknowledgedMessages() method, Consumer every time epoch + 1.
But broker may be not increment. In the following code.

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java

Lines 310 to 322 in 4f1e39b

	if (!havePendingRead) {
	if (consumerEpoch > consumer.getConsumerEpoch()) {
	consumer.setConsumerEpoch(consumerEpoch);
	}
	cursor.rewind();
	if (log.isDebugEnabled()) {
	log.debug("[{}-{}] Cursor rewinded, redelivering unacknowledged messages. ", name, consumer);
	}
	readMoreEntries(consumer);
	} else {
	log.info("[{}-{}] Ignoring reDeliverUnAcknowledgedMessages: cancelPendingRequest on cursor failed", name,
	consumer);
	}

This unit test is caused by the inconsistency between the epoch of the consumer and the broker.

@codelipenghui @congbobo184 Can you take a look?

[codelipenghui]

Nice catch

@congbobo184 Could you please take a look?