[Bug] Turn off allowAutoTopicCreation, then turn on transactionCoordinatorEnabled, unable to send, consume messages

[tianshimoyi]

Search before asking

• I searched in the issues and found nothing similar.

Version

streamnative/pulsar-all/pulsar-all:2.9.4.1

Minimal reproduce step

• allowAutoTopicCreation = false
• transactionCoordinatorEnabled = true
• create a topic
• produce message

What did you expect to see?

produce message successfully

What did you see instead?

Anything else?

Then I opened the topic automatic creation configuration for a certain namespace, and the message was sent successfully！

Are you willing to submit a PR?

• I'm willing to submit a PR!

[tianshimoyi]

@congbobo184

[congbobo184]

@tianshimoyi #14949 has fixed this problem, please use 2.9.3 or above version

[tianshimoyi]

@congbobo184 My image is streamnative/pulsar-all/pulsar-all:2.9.4.1, shouldn't it be higher than version 2.9.3?

[congbobo184]

@tianshimoyi hi, are you set systemTopicEnabled=false, if disable system topic will throw this exception

[tianshimoyi]

@congbobo184 I set systemTopicEnabled=true, topicLevelPoliciesEnabled=true, non-transactional messages can be sent successfully, but transactional messages cannot

[tianshimoyi]

@congbobo184 I set up a new pulsar cluster. The two clusters share a zk cluster. There is no problem with the new pulsar cluster. The configuration is the same. What causes the above problems?

[congbobo184]

@tianshimoyi do you have the broker log can be shared for me?

[congbobo184]

do you have the full log? this picture seem can't analyze any problems

[tianshimoyi]

@congbobo184
2023-02-08T06:34:12,402+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of public/default/0x00000000_0x10000000
2023-02-08T06:34:12,402+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for public/default/0x00000000_0x10000000
2023-02-08T06:34:12,402+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[public/default/0x00000000_0x10000000]
2023-02-08T06:34:12,403+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.35.166:36022
2023-02-08T06:34:15,597+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40676
2023-02-08T06:34:15,598+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40680
2023-02-08T06:34:15,599+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40682
2023-02-08T06:34:15,599+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40684
2023-02-08T06:34:15,600+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40688
2023-02-08T06:34:15,600+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40690
2023-02-08T06:34:15,601+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40692
2023-02-08T06:34:15,601+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40696
2023-02-08T06:34:15,602+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40698
2023-02-08T06:34:15,602+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40700
2023-02-08T06:34:15,602+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40702
2023-02-08T06:34:15,603+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40706
2023-02-08T06:34:15,603+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40708
2023-02-08T06:34:15,604+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40710
2023-02-08T06:34:15,605+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40712
2023-02-08T06:34:15,605+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.22:40714
2023-02-08T06:34:15,611+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x50000000_0x60000000
2023-02-08T06:34:15,611+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x50000000_0x60000000
2023-02-08T06:34:15,611+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x50000000_0x60000000]
2023-02-08T06:34:15,612+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,612+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,612+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x80000000_0x90000000]
2023-02-08T06:34:15,612+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,612+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,612+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xf0000000_0xffffffff]
2023-02-08T06:34:15,612+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,612+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,612+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x80000000_0x90000000]
2023-02-08T06:34:15,613+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x10000000_0x20000000
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x10000000_0x20000000
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x10000000_0x20000000]
2023-02-08T06:34:15,613+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x60000000_0x70000000
2023-02-08T06:34:15,613+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x60000000_0x70000000
2023-02-08T06:34:15,613+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x60000000_0x70000000]
2023-02-08T06:34:15,613+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xf0000000_0xffffffff]
2023-02-08T06:34:15,613+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x10000000_0x20000000
2023-02-08T06:34:15,613+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x10000000_0x20000000
2023-02-08T06:34:15,613+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x10000000_0x20000000]
2023-02-08T06:34:15,613+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xf0000000_0xffffffff
2023-02-08T06:34:15,613+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xf0000000_0xffffffff]
2023-02-08T06:34:15,614+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x80000000_0x90000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x80000000_0x90000000]
2023-02-08T06:34:15,614+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xb0000000_0xc0000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xb0000000_0xc0000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xb0000000_0xc0000000]
2023-02-08T06:34:15,614+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xc0000000_0xd0000000
2023-02-08T06:34:15,614+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xc0000000_0xd0000000
2023-02-08T06:34:15,614+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xc0000000_0xd0000000]
2023-02-08T06:34:15,614+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0x20000000_0x30000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0x20000000_0x30000000
2023-02-08T06:34:15,614+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0x20000000_0x30000000]
2023-02-08T06:34:15,615+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40676
2023-02-08T06:34:15,629+0000 [pulsar-2-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xb0000000_0xc0000000
2023-02-08T06:34:15,629+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xb0000000_0xc0000000
2023-02-08T06:34:15,629+0000 [pulsar-2-1] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xb0000000_0xc0000000]
2023-02-08T06:34:15,629+0000 [pulsar-2-2] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - 0 brokers being considered for assignment of pulsar/system/0xc0000000_0xd0000000
2023-02-08T06:34:15,629+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - No broker is available for pulsar/system/0xc0000000_0xd0000000
2023-02-08T06:34:15,629+0000 [pulsar-2-2] WARN org.apache.pulsar.broker.namespace.NamespaceService - Load manager didn't return any available broker. Returning empty result to lookup. NamespaceBundle[pulsar/system/0xc0000000_0xd0000000]
2023-02-08T06:34:15,629+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40680
2023-02-08T06:34:15,631+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40682
2023-02-08T06:34:15,632+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40684
2023-02-08T06:34:15,634+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40688
2023-02-08T06:34:15,635+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40690
2023-02-08T06:34:15,636+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40692
2023-02-08T06:34:15,638+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40696
2023-02-08T06:34:15,639+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40700
2023-02-08T06:34:15,640+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40698
2023-02-08T06:34:15,641+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40702
2023-02-08T06:34:15,643+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40706
2023-02-08T06:34:15,648+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40708
2023-02-08T06:34:15,649+0000 [pulsar-io-6-2] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40710
2023-02-08T06:34:15,651+0000 [pulsar-io-6-3] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40712
2023-02-08T06:34:15,652+0000 [pulsar-io-6-4] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /10.0.49.22:40714
2023-02-08T06:34:18,188+0000 [pulsar-web-43-5] INFO org.eclipse.jetty.server.RequestLog - 10.0.72.26 - - [08/Feb/2023:06:34:18 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/1.18" 1
2023-02-08T06:34:18,441+0000 [pulsar-io-6-1] INFO org.apache.pulsar.broker.service.ServerCnx - New connection from /10.0.49.165:38526
2023-02-08T06:34:18,441+0000 [pulsar-io-6-1] WARN org.apache.pulsar.broker.service.ServerCnx - [/10.0.49.165:38526] Got exception io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 5253120: 1195725860 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:507)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:493)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:377)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:423)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)

[congbobo184]

seems like brokers are all down, load balancer can't find an available broker.
can you test:

1. create a new topic
2. send messages without transaction
   what the result of this case?

[tianshimoyi]

@congbobo184 Yes, all broker nodes are unavailable, and restarting all broker nodes will not work. Do you know why this is? Or how to fix this problem?

[congbobo184]

may be starting two clusters in the same zk will produce some problems, It's better to check the start up log to locate the problem.