[Bug] Test FAILURE! - in org.apache.pulsar.io.PulsarFunctionTlsTest

[dave2wave]

Search before asking

• I searched in the issues and found nothing similar.

Version

Pulsar Master branch

Minimal reproduce step

Maven build with tests

What did you expect to see?

No errors

What did you see instead?

Logs show:

11:22:40  [ERROR] Tests run: 6, Failures: 3, Errors: 0, Skipped: 3, Time elapsed: 73.847 s <<< FAILURE! - in org.apache.pulsar.io.PulsarFunctionTlsTest
11:22:40  [ERROR] org.apache.pulsar.io.PulsarFunctionTlsTest.setup[public void org.apache.pulsar.io.PulsarFunctionTlsTest.testAuthorization()](0)  Time elapsed: 73.276 s  <<< FAILURE!
11:22:40  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /ledgers/available/null:0
11:22:40  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
11:22:40  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:22:40  	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1869)
11:22:40  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:303)
11:22:40  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:414)
11:22:40  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:422)
11:22:40  	at org.apache.pulsar.io.PulsarFunctionTlsTest.setup(PulsarFunctionTlsTest.java:108)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:22:40  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:22:40  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:22:40  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:22:40  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:22:40  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:22:40  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:22:40  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:22:40  	at org.testng.TestRunner.run(TestRunner.java:602)
11:22:40  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:22:40  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:22:40  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:22:40  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:22:40  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:22:40  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:22:40  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:22:40  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:22:40  	at org.testng.TestNG.runSuites(TestNG.java:1099)
11:22:40  	at org.testng.TestNG.run(TestNG.java:1067)
11:22:40  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:22:40  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:22:40  
11:22:40  [ERROR] org.apache.pulsar.io.PulsarFunctionTlsTest.shutdown  Time elapsed: 0.017 s  <<< FAILURE!
11:22:40  java.lang.NullPointerException: Cannot invoke "org.apache.pulsar.client.admin.PulsarAdmin.close()" because "this.functionAdmin" is null
11:22:40  	at org.apache.pulsar.io.PulsarFunctionTlsTest.shutdown(PulsarFunctionTlsTest.java:176)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:22:40  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:22:40  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:22:40  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:22:40  	at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:613)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:22:40  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:22:40  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:22:40  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:22:40  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:22:40  	at org.testng.TestRunner.run(TestRunner.java:602)
11:22:40  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:22:40  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:22:40  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:22:40  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:22:40  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:22:40  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:22:40  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:22:40  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:22:40  	at org.testng.TestNG.runSuites(TestNG.java:1099)
11:22:40  	at org.testng.TestNG.run(TestNG.java:1067)
11:22:40  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:22:40  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:22:40  
11:22:40  [ERROR] org.apache.pulsar.io.PulsarFunctionTlsTest.shutdown  Time elapsed: 0.001 s  <<< FAILURE!
11:22:40  java.lang.NullPointerException: Cannot invoke "org.apache.pulsar.client.admin.PulsarAdmin.close()" because "this.functionAdmin" is null
11:22:40  	at org.apache.pulsar.io.PulsarFunctionTlsTest.shutdown(PulsarFunctionTlsTest.java:176)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:22:40  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:22:40  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:22:40  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:22:40  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:22:40  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:22:40  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:22:40  	at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:613)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:22:40  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:22:40  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:22:40  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:22:40  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:22:40  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:22:40  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:22:40  	at org.testng.TestRunner.run(TestRunner.java:602)
11:22:40  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:22:40  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:22:40  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:22:40  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:22:40  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:22:40  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:22:40  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:22:40  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:22:40  	at org.testng.TestNG.runSuites(TestNG.java:1099)
11:22:40  	at org.testng.TestNG.run(TestNG.java:1067)
11:22:40  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:22:40  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:22:40  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:22:40  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:22:40  

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[dave2wave]

Also saw this a little later with a Session Expired.

11:25:10  [ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 11, Time elapsed: 290.474 s <<< FAILURE! - in org.apache.pulsar.io.PulsarFunctionE2ETest
11:25:10  [ERROR] org.apache.pulsar.io.PulsarFunctionE2ETest.setup[public void org.apache.pulsar.io.PulsarFunctionE2ETest.testE2EPulsarFunctionWithNarFile() throws java.lang.Exception](0)  Time elapsed: 51.4 s  <<< FAILURE!
11:25:10  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /ledgers/available/null:0
11:25:10  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
11:25:10  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:25:10  	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1869)
11:25:10  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:303)
11:25:10  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:414)
11:25:10  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:422)
11:25:10  	at org.apache.pulsar.io.AbstractPulsarE2ETest.setup(AbstractPulsarE2ETest.java:110)
11:25:10  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:25:10  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:25:10  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:25:10  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:25:10  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:25:10  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:25:10  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:25:10  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:25:10  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:25:10  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
11:25:10  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:25:10  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:25:10  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:25:10  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:25:10  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:25:10  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:25:10  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:25:10  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:25:10  	at org.testng.TestRunner.run(TestRunner.java:602)
11:25:10  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:25:10  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:25:10  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:25:10  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:25:10  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:25:10  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:25:10  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:25:10  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:25:10  	at org.testng.TestNG.runSuites(TestNG.java:1099)
11:25:10  	at org.testng.TestNG.run(TestNG.java:1067)
11:25:10  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:25:10  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:25:10  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:25:10  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:25:10  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:25:10  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:25:10  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:25:10  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:25:10  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:25:10  

[dave2wave]

And

11:26:42  [ERROR] org.apache.pulsar.broker.service.ReplicatorRemoveClusterTest.setup  Time elapsed: 61.914 s  <<< FAILURE!
11:26:42  org.testng.internal.invokers.InvokeMethodRunnable$TestNGRuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /ledgers/available/null:0
11:26:42  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:56)
11:26:42  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
11:26:42  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
11:26:42  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
11:26:42  	at java.base/java.lang.Thread.run(Thread.java:833)
11:26:42  Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /ledgers/available/null:0
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:26:42  	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1869)
11:26:42  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies(LocalBookkeeperEnsemble.java:303)
11:26:42  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:414)
11:26:42  	at org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.start(LocalBookkeeperEnsemble.java:422)
11:26:42  	at org.apache.pulsar.broker.service.ReplicatorTestBase.setup(ReplicatorTestBase.java:155)
11:26:42  	at org.apache.pulsar.broker.service.ReplicatorRemoveClusterTest.setup(ReplicatorRemoveClusterTest.java:56)
11:26:42  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:26:42  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:26:42  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:26:42  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:26:42  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:26:42  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
11:26:42  	... 6 more
11:26:42  
11:26:42  [ERROR] org.apache.pulsar.broker.service.ReplicatorRemoveClusterTest.beforeMethod[public void org.apache.pulsar.broker.service.ReplicatorRemoveClusterTest.testRemoveClusterFromNamespace() throws java.lang.Exception](0)  Time elapsed: 63.365 s  <<< FAILURE!
11:26:42  org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: 
11:26:42  
11:26:42   --- An unexpected error occurred in the server ---
11:26:42  
11:26:42  Message: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  
11:26:42  Stacktrace:
11:26:42  
11:26:42  org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$existsFromStore$9(ZKMetadataStore.java:353)
11:26:42  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
11:26:42  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
11:26:42  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
11:26:42  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:26:42  	at java.base/java.lang.Thread.run(Thread.java:833)
11:26:42  Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
11:26:42  	... 8 more
11:26:42  
11:26:42  	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
11:26:42  	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
11:26:42  	at org.apache.pulsar.client.admin.internal.NamespacesImpl.removeBacklogQuota(NamespacesImpl.java:661)
11:26:42  	at org.apache.pulsar.client.admin.Namespaces.removeBacklogQuota(Namespaces.java:1709)
11:26:42  	at org.apache.pulsar.broker.service.ReplicatorRemoveClusterTest.beforeMethod(ReplicatorRemoveClusterTest.java:48)
11:26:42  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:26:42  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
11:26:42  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:26:42  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
11:26:42  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
11:26:42  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
11:26:42  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
11:26:42  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
11:26:42  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
11:26:42  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
11:26:42  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
11:26:42  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
11:26:42  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
11:26:42  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
11:26:42  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
11:26:42  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
11:26:42  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
11:26:42  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
11:26:42  	at org.testng.TestRunner.run(TestRunner.java:602)
11:26:42  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
11:26:42  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
11:26:42  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
11:26:42  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
11:26:42  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
11:26:42  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
11:26:42  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
11:26:42  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
11:26:42  	at org.testng.TestNG.runSuites(TestNG.java:1099)
11:26:42  	at org.testng.TestNG.run(TestNG.java:1067)
11:26:42  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
11:26:42  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
11:26:42  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
11:26:42  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
11:26:42  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
11:26:42  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
11:26:42  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
11:26:42  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
11:26:42  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
11:26:42  	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: 
11:26:42   --- An unexpected error occurred in the server ---
11:26:42  
11:26:42  Message: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  
11:26:42  Stacktrace:
11:26:42  
11:26:42  org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$existsFromStore$9(ZKMetadataStore.java:353)
11:26:42  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
11:26:42  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
11:26:42  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
11:26:42  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
11:26:42  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:26:42  	at java.base/java.lang.Thread.run(Thread.java:833)
11:26:42  Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
11:26:42  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
11:26:42  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
11:26:42  	... 8 more
11:26:42  
11:26:42  		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
11:26:42  		at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237)
11:26:42  		at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
11:26:42  		at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
11:26:42  		at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
11:26:42  		at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
11:26:42  		at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
11:26:42  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
11:26:42  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
11:26:42  		at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
11:26:42  		at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
11:26:42  		at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
11:26:42  		at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
11:26:42  		at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
11:26:42  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
11:26:42  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
11:26:42  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
11:26:42  		at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
11:26:42  		at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
11:26:42  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
11:26:42  		at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
11:26:42  		at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
11:26:42  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:26:42  		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:26:42  		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
11:26:42  		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
11:26:42  		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
11:26:42  		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
11:26:42  		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
11:26:42  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
11:26:42  		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
11:26:42  		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
11:26:42  		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
11:26:42  		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
11:26:42  		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
11:26:42  		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
11:26:42  		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
11:26:42  		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
11:26:42  		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
11:26:42  		at java.base/java.lang.Thread.run(Thread.java:833)
11:26:42  	Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$existsFromStore$9(ZKMetadataStore.java:353)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:134)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 8 more\n"}
11:26:42  		at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
11:26:42  		at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
11:26:42  		... 54 more
11:26:42  Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$existsFromStore$9(ZKMetadataStore.java:353)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /admin/flags/policies-readonly\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:134)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 8 more\n"}]
11:26:42  

[dave2wave]

Are some of the tests expecting to only run as github workflows?

[dave2wave]

I no longer see these particular errors