Persistent topic is always temporarily unavailable

[codelipenghui]

Describe the bug
Here is the error log at broker side:

 23:29:38.178 [pulsar-io-22-19] WARN  org.apache.pulsar.broker.service.ServerCnx - [/10.1.1.101:54749][persistent://swap_okex_prod/order_event/66][SUB_PP_EXCHANGE] Failed to create consumer: Topic is temporarily unavailable
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException: Topic is temporarily unavailable
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) ~[?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:984) ~[?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) ~[?:1.8.0_121]
        at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:680) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) ~[?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) ~[?:1.8.0_121]
        at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$15(ServerCnx.java:633) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614) [?:1.8.0_121]
        at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983) [?:1.8.0_121]
        at org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:622) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
        at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:199) [org.apache.pulsar-pulsar-common-2.4.1.jar:2.4.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
 

 23:29:48.235 [ForkJoinPool.commonPool-worker-7] WARN  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://swap_okex_prod/order_event/66] Attempting to add producer to a fenced topic
23:29:48.235 [ForkJoinPool.commonPool-worker-7] ERROR org.apache.pulsar.broker.service.ServerCnx - [/10.1.1.97:39705] Failed to add producer to topic persistent://swap_okex_prod/order_event/66: Topic is temporarily unavailable

To Reproduce
Difficult to reproduce

[aahmed-se]

is this for a new consumer ?

[codelipenghui]

@aahmed-se

is this for a new consumer ?

both new producer and new consumer, and lasted for a few minutes, restart the broker can resolve.

[TC-oKozlov]

We are getting a similar issue from our custom client code when running a load test from 10 concurrent producer processes. In our setup we create a new producer before sending each message, send message, and close producer. We start getting "topic temporarily unavailable" after about 8000 producers / messages were created. This is runs through Pulsar Proxy ->Broker deployed on kubernetes. It seems like Pulsar starts having an issue when too many producers are created over a short period of time. With our load test we are able to reproduce it pretty consistently:

Re reproducing:

1. start several processes in parallel (in our case 10)
2. for each process: create producer send message, close producer
   We started 100k messages per process (1mil total), but started getting this error only after about 8k msgs were sent.

[tuteng]

Can you give me a test code? It seems that it is difficult for me to reproduce this problem locally. including some configurations of the cluster. @TC-oKozlov