Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Geo-replication failed when enabling partitioned topic auto-creation #6840

Closed
codelipenghui opened this issue Apr 29, 2020 · 0 comments · Fixed by #6846
Closed

Geo-replication failed when enabling partitioned topic auto-creation #6840

codelipenghui opened this issue Apr 29, 2020 · 0 comments · Fixed by #6846
Assignees
@codelipenghui
Labels
release/2.5.2 type/bug The PR fixed a bug or issue reported a bug
Milestone
2.6.0

Comments

@codelipenghui
Copy link
Contributor

Describe the bug
Geo-replication failed when enabling partitioned topic auto-creation

To Reproduce
Steps to reproduce the behavior:

  1. Start a geo Pulsar cluster
  2. Enable the partitioned topic auto-creation(allowAutoTopicCreationType=partitioned)
  3. Create a namespace with 2 clusters
  4. Start a producer to produce messages to a topic(don't create it by pulsar-admin)
  5. The error occurs at a local cluster
16:19:02.142 [pulsar-io-22-24] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://geo/default/pub-test-partition-2] [pulsar.repl.pulsar-cluster-a] Failed to create producer: persistent://geo/default/pub-test-partition-2 Failed to start replicator pulsar-cluster-a
16:19:02.142 [pulsar-io-22-24] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://geo/default/pub-test-partition-2] [pulsar.repl.pulsar-cluster-a] Could not get connection to broker: persistent://geo/default/pub-test-partition-2 Failed to start replicator pulsar-cluster-a -- Will try again in 3.13 s
16:19:02.355 [pulsar-timer-42-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://geo/default/pub-test-partition-1] [pulsar.repl.pulsar-cluster-a] Reconnecting after connection was closed
16:19:02.357 [pulsar-io-22-8] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://geo/default/pub-test-partition-1] [pulsar.repl.pulsar-cluster-a] Creating producer on cnx [id: 0xbed80cce, L:/192.168.1.102:54848 - R:/192.168.1.102:6660]
16:19:02.360 [pulsar-io-22-18] WARN  org.apache.pulsar.client.impl.ClientCnx - [id: 0xbed80cce, L:/192.168.1.102:54848 - R:/192.168.1.102:6660] Received error from server: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a
16:19:02.360 [pulsar-io-22-18] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://geo/default/pub-test-partition-1] [pulsar.repl.pulsar-cluster-a] Failed to create producer: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a
16:19:02.360 [pulsar-io-22-18] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://geo/default/pub-test-partition-1] [pulsar.repl.pulsar-cluster-a] Could not get connection to broker: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a -- Will try again in 2.919 s

16:18:59.839 [pulsar-ordered-OrderedExecutor-1-0] WARN  org.apache.pulsar.broker.service.BrokerService - Replication or dedup check failed. Removing topic from topics list persistent://geo/default/pub-test-partition-0, java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-0 Failed to start replicator pulsar-cluster-a
16:18:59.839 [pulsar-ordered-OrderedExecutor-1-0] ERROR org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.102:54867] Failed to create topic persistent://geo/default/pub-test-partition-0
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-0 Failed to start replicator pulsar-cluster-a
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.andTree(CompletableFuture.java:1321) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.allOf(CompletableFuture.java:2238) ~[?:1.8.0_241]
	at org.apache.pulsar.common.util.FutureUtil.waitForAll(FutureUtil.java:37) ~[org.apache.pulsar-pulsar-common-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkReplication(PersistentTopic.java:1103) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.BrokerService$2.openLedgerComplete(BrokerService.java:892) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:341) ~[org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_241]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen(ManagedLedgerFactoryImpl.java:340) ~[org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.BrokerService.lambda$createPersistentTopic$25(BrokerService.java:885) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_241]
	at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$34(BrokerService.java:1036) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-0 Failed to start replicator pulsar-cluster-a
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.startReplicator(PersistentTopic.java:1180) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkReplication(PersistentTopic.java:1086) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	... 18 more
16:19:00.081 [pulsar-io-22-3] INFO  org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.102:54857][persistent://geo/default/pub-test-partition-1] Creating producer. producerId=3
16:19:00.081 [pulsar-ordered-OrderedExecutor-2-0] INFO  org.apache.pulsar.broker.service.AbstractTopic - Disabling publish throttling for persistent://geo/default/pub-test-partition-1
16:19:00.082 [pulsar-ordered-OrderedExecutor-2-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://geo/default/pub-test-partition-1] There are no replicated subscriptions on the topic
16:19:00.082 [pulsar-ordered-OrderedExecutor-2-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://geo/default/pub-test-partition-1] Starting replicator to remote: pulsar-cluster-a
16:19:00.082 [pulsar-ordered-OrderedExecutor-2-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://geo/default/pub-test-partition-1] Replicator startup failed due to partitioned-topic pulsar-cluster-a
16:19:00.082 [pulsar-ordered-OrderedExecutor-2-0] WARN  org.apache.pulsar.broker.service.BrokerService - Replication or dedup check failed. Removing topic from topics list persistent://geo/default/pub-test-partition-1, java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a
16:19:00.082 [pulsar-ordered-OrderedExecutor-2-0] ERROR org.apache.pulsar.broker.service.ServerCnx - [/192.168.1.102:54857] Failed to create topic persistent://geo/default/pub-test-partition-1
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.andTree(CompletableFuture.java:1321) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.allOf(CompletableFuture.java:2238) ~[?:1.8.0_241]
	at org.apache.pulsar.common.util.FutureUtil.waitForAll(FutureUtil.java:37) ~[org.apache.pulsar-pulsar-common-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkReplication(PersistentTopic.java:1103) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.BrokerService$2.openLedgerComplete(BrokerService.java:892) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$7(ManagedLedgerFactoryImpl.java:341) ~[org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_241]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.asyncOpen(ManagedLedgerFactoryImpl.java:340) ~[org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.BrokerService.lambda$createPersistentTopic$25(BrokerService.java:885) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_241]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_241]
	at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$34(BrokerService.java:1036) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [org.apache.pulsar-managed-ledger-2.5.1.jar:2.5.1]
	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$NamingException: persistent://geo/default/pub-test-partition-1 Failed to start replicator pulsar-cluster-a
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.startReplicator(PersistentTopic.java:1180) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.checkReplication(PersistentTopic.java:1086) ~[org.apache.pulsar-pulsar-broker-2.5.1.jar:2.5.1]
	... 18 more

Expected behavior
Geo-replication should work well

Additional context
Version 2.5.1
@codelipenghui codelipenghui added the type/bug The PR fixed a bug or issue reported a bug label Apr 29, 2020
@sijie sijie mentioned this issue Apr 29, 2020
Closed
@codelipenghui codelipenghui self-assigned this Apr 29, 2020
@codelipenghui codelipenghui added this to the 2.6.0 milestone Apr 29, 2020
@codelipenghui codelipenghui mentioned this issue Apr 29, 2020
Merged
jiazhai pushed a commit that referenced this issue Apr 30, 2020
@codelipenghui
Avoid creating partitioned topic for partition name (#6846)
bb41702 
Fixes #6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
gaoran10 pushed a commit to gaoran10/pulsar that referenced this issue May 5, 2020
@codelipenghui @gaoran10
Avoid creating partitioned topic for partition name (apache#6846)
52b58b5 
Fixes apache#6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
gaoran10 pushed a commit to gaoran10/pulsar that referenced this issue May 5, 2020
@codelipenghui @gaoran10
Avoid creating partitioned topic for partition name (apache#6846)
f473d81 
Fixes apache#6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
addisonj pushed a commit to instructure/pulsar that referenced this issue May 7, 2020
@codelipenghui
Avoid creating partitioned topic for partition name (apache#6846)
c3f97d9 
Fixes apache#6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
jiazhai pushed a commit that referenced this issue May 8, 2020
@codelipenghui @jiazhai
Avoid creating partitioned topic for partition name (#6846)
a2d5d5b 
Fixes #6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
(cherry picked from commit bb41702)
Huanli-Meng pushed a commit to Huanli-Meng/pulsar that referenced this issue May 27, 2020
@codelipenghui @Huanli-Meng
Avoid creating partitioned topic for partition name (apache#6846)
95c16a5 
Fixes apache#6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this issue Aug 24, 2020
@codelipenghui @huangdx0726
Avoid creating partitioned topic for partition name (apache#6846)
3125e00 
Fixes apache#6840

Motivation
Avoid creating partitioned topic for partition name

Verifying this change
New unit test added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codelipenghui codelipenghui

Labels
release/2.5.2 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
2.6.0
Development

Successfully merging a pull request may close this issue.

Avoid creating partitioned topic for partition name codelipenghui/incubator-pulsar
1 participant
@codelipenghui