Java client connection lookup timed out, 7 lookup request timedout after ms 30000

[jiangmincong]

Expected behavior

2020-11-06 14:13:16,906 | WARN | [pulsar-client-io-1-1] | Failed subscription for createPartitionedConsumer: persistent://tenant/namespace/topic_test 30, e:{} | org.apache.pulsar.client.impl.MultiTopicsConsumerImpl (MultiTopicsConsumerImpl.java:775)
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 7 lookup request timedout after ms 30000
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1284)
at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1270)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.pulsar.client.impl.ConsumerImpl.connectionFailed(ConsumerImpl.java:710)
at org.apache.pulsar.client.impl.ConnectionHandler.handleConnectionError(ConnectionHandler.java:78)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.pulsar.client.impl.BinaryProtoLookupService.lambda$null$3(BinaryProtoLookupService.java:147)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.apache.pulsar.client.impl.ClientCnx.lambda$addPendingLookupRequests$12(ClientCnx.java:537)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:150)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Actual behavior

Tell us what happens instead

Steps to reproduce

How can we reproduce the issue

System configuration

**Pulsar 2.6.1

[jiangmincong]

Our test found that this situation is random. Lookup timeouts are rare when there are not many consumers, and there is a high probability when there are many consumers. The situation we tested is 30 topic partitions, continuous When multiple programs are started, the probability of the later startup program reporting lookup timeout is significantly higher than that of the previous startup, causing the consumer program to hang

[jiazhai]

@jiangmincong This seems more related to the issue #8533. And @reswqa is helping on this issue.

[jiazhai]

@jiangmincong What is the zookeeper side stats when the lookup hang happens? Is Zookeeper in a busy status?

[jiangmincong]

Hi， I looked at the monitoring and it seemed that the zookeeper status was not very busy during the abnormal time.

[jiangmincong]

From the JVM monitoring, there seems to be no exception

[jiangmincong]

This exception occurs when there are two different versions (2.5.0/2.6.1) of client consumption or production data at the same time. After replacing the topic, this exception does not appear. I will close this issue.