[Pulsar-Broker] consistent test failure - Namespace already exists

[devinbost]

I'm now getting this issue consistently when running builds from Github CI, and it's also happening consistently when I run the test locally. The test fails with one of two different exceptions. Either this one:

[ERROR] Tests run: 20, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 345.803 s <<< FAILURE! - in org.apache.pulsar.broker.service.ReplicatorTest
464[ERROR] testReplicatedCluster(org.apache.pulsar.broker.service.ReplicatorTest) Time elapsed: 0.023 s <<< FAILURE!
465org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Namespace already exists

or this one:

org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available

	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:691)
	at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93)
	at org.apache.pulsar.broker.service.ReplicatorTest.testReplicatedCluster(ReplicatorTest.java:834)
	. . . 

I've actually experienced intermittent (random) issues in production that have given the message "Namespace already exists," but we never captured broker logs when this has happened.
It's not clear if those problems are related to this issue.

The logs can be seen here: https://github.com/apache/pulsar/pull/6202/checks?check_run_id=431093396

I've also attached the logs.

logs_14078.zip

[devinbost]

When I run this test locally, I'm getting this exception:

org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available

	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:691)
	at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93)
	at org.apache.pulsar.broker.service.ReplicatorTest.testReplicatedCluster(ReplicatorTest.java:834)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

I've attached the test log and the test runner log from reproducing this issue locally:

test_log.txt
test_runner_log.txt

This issue consistently occurs when I run the test from master, and it's also blocking my PR #6202

[devinbost]

@sijie FYI.

[sijie]

Namespace already exists usually because tests are re-run. So you might consider improving the test with randomized names.

[tisonkun]

Closed as answered.