[FlakyTest]Try to fix flaky test ReplicatorTest.testReplicatorOnPartitionedTopic. by MarvinCai · Pull Request #9659 · apache/pulsar

MarvinCai · 2021-02-21T08:09:49Z

Motivation

Not sure why state is not cleaned up but exception is indicating metadata of namespace the test trying to create already exist so use a different namespace name for different test run should able to fix it.

Modifications

Append System.nanoTime to the namespace used to avoid being affect by previous state. This seems working fine for other test cases in the same test.

Try to make the test stable by append System.nanoTime to namespace it creates to avoid "Namespace already exists" exception.

eolivelli

Usually that error is due to the test retry policy and it hides the real error.

This fix cannot be enough to fix the flakyness.

Do you have the real error in the test? That is the logs of the first run that failed?

MarvinCai · 2021-02-21T09:56:16Z

Usually that error is due to the test retry policy and it hides the real error.

This fix cannot be enough to fix the flakyness.

Do you have the real error in the test? That is the logs of the first run that failed?

@eolivelli Sample stack trace is

org.apache.pulsar.client.admin.PulsarAdminException$ConflictException: Namespace already exists
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:219)
	at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:129)
	at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:839)
	at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:820)
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:210)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$3(AsyncHttpConnector.java:251)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
	at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:938)
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:921)
	at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:77)
	... 54 more

Make the namespace name unique for different execution/retry should able to get rid of this exception.

eolivelli

I believe that if we want to go this way we should have a more consistent way of generating unique topic names.

Can we create a general purpose function that creates an unique topic name? IIRC there is something like that somewhere else is some suite

315157973 · 2021-02-22T12:05:48Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ReplicatorTest.java

        log.info("--- Starting ReplicatorTest::{} --- ", methodName);

-        final String namespace = "pulsar/partitionedNs-" + isPartitionedTopic;
+        final String namespace = "pulsar/partitionedNs-" + isPartitionedTopic + "-" + System.nanoTime();;


1）I searched the code, but I didn’t see a namespace with the same name in other places
2）There are two ";"

fixed the duplicate ";"
It could be namespace left from other test run, the zookeeper used for test will have a temp dir to store data with same name. It could be really hard to debug in github action VM but make namespace unique across different test run should solve the issue for us.

eolivelli · 2021-02-22T12:25:47Z

We could use something like
BrokerTestBase#newTopicName

making it a static Utility method (and possibly extracting it to a separate file

https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerTestBase.java#L93

MarvinCai · 2021-02-23T07:00:43Z

@eolivelli Updated per you comment.

eolivelli

What about using the new method in BrokerTestBase, or better to remove the newTopicName in BrokerTestBase ?

pulsar-broker/src/test/java/org/apache/pulsar/broker/BrokerTestUtil.java

…o in ReplicatorTest to use BrokerTestUtil.newUniqueName.

MarvinCai · 2021-02-25T04:01:32Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-25T05:20:17Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-25T06:07:43Z

/pulsarbot run-failure-checks

eolivelli

I came late to the party.

btw LGTM

thank you @MarvinCai

…tionedTopic. (apache#9659) Fixes apache#9457 Not sure why state is not cleaned up but exception is indicating metadata of namespace the test trying to create already exist so use a different namespace name for different test run should able to fix it. Append System.nanoTime to the namespace used to avoid being affect by previous state. This seems working fine for other test cases in the same test. (cherry picked from commit 738ec6a)

Try to fix flaky test ReplicatorTest.testReplicatorOnPartitionedTopic.

52740f7

Try to make the test stable by append System.nanoTime to namespace it creates to avoid "Namespace already exists" exception.

eolivelli requested changes Feb 21, 2021

View reviewed changes

sijie approved these changes Feb 22, 2021

View reviewed changes

sijie assigned MarvinCai Feb 22, 2021

sijie added the type/flaky-tests label Feb 22, 2021

sijie added this to the 2.8.0 milestone Feb 22, 2021

eolivelli requested changes Feb 22, 2021

View reviewed changes

315157973 reviewed Feb 22, 2021

View reviewed changes

Address comments.

bc6bdff

eolivelli reviewed Feb 23, 2021

View reviewed changes

pulsar-broker/src/test/java/org/apache/pulsar/broker/BrokerTestUtil.java Outdated Show resolved Hide resolved

MarvinCai added 2 commits February 24, 2021 14:44

Address comments.

9c1c2ee

Convert all NanoTime and CurrentTimeMillis based random name generati…

7664474

…o in ReplicatorTest to use BrokerTestUtil.newUniqueName.

MarvinCai requested a review from 315157973 February 25, 2021 02:14

codelipenghui approved these changes Feb 25, 2021

View reviewed changes

codelipenghui merged commit 738ec6a into apache:master Feb 25, 2021

eolivelli reviewed Feb 25, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FlakyTest]Try to fix flaky test ReplicatorTest.testReplicatorOnPartitionedTopic.#9659

[FlakyTest]Try to fix flaky test ReplicatorTest.testReplicatorOnPartitionedTopic.#9659
codelipenghui merged 4 commits intoapache:masterfrom
MarvinCai:zxc/replicator-test-fix

MarvinCai commented Feb 21, 2021 •

edited

Loading

Uh oh!

eolivelli left a comment

Uh oh!

MarvinCai commented Feb 21, 2021

Uh oh!

eolivelli left a comment

Uh oh!

315157973 Feb 22, 2021

Uh oh!

MarvinCai Feb 23, 2021

Uh oh!

eolivelli commented Feb 22, 2021

Uh oh!

MarvinCai commented Feb 23, 2021

Uh oh!

eolivelli left a comment

Uh oh!

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

eolivelli left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

MarvinCai commented Feb 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

MarvinCai commented Feb 21, 2021

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

315157973 Feb 22, 2021

Choose a reason for hiding this comment

Uh oh!

MarvinCai Feb 23, 2021

Choose a reason for hiding this comment

Uh oh!

eolivelli commented Feb 22, 2021

Uh oh!

MarvinCai commented Feb 23, 2021

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

MarvinCai commented Feb 25, 2021

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MarvinCai commented Feb 21, 2021 •

edited

Loading