Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transactions: if you enable the TransactionCoordinator force delete topic leaves the broker in bad state #12754

Closed
eolivelli opened this issue Nov 11, 2021 · 3 comments · Fixed by #12758
Closed

Transactions: if you enable the TransactionCoordinator force delete topic leaves the broker in bad state #12754

eolivelli opened this issue Nov 11, 2021 · 3 comments · Fixed by #12758
Assignees
@congbobo184
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@eolivelli
Copy link
Contributor

Describe the bug
I found this problem while testing Pulsar 2.9.0rc3 and running the unit tests of Pulsar JMS (http://github.com/datastax/pulsar-jms).

When you enable the TransactionCoordinator deleting a topic forcefully leaves the broker in such a bad state that it is no more possible to consume data from other topics.

To Reproduce

  • Start one Broker and setup Transactions
  • Create one topic in public/default
  • Create one subscription on the topic
  • Delete the topic with "force" = true
  • Create another topic with a different name (in the same namespace)
  • Create one subscription on the second topic
  • Produce a message to the second topic
  • Try to receive a message from the second topic, Consumer.receive() will hang forever

Expected behavior
The second topic works well

Additional context
The problem is present in 2.9.0rc3 but NOT in Pulsar 2.8.1
Running on JDK8/Mac
@eolivelli eolivelli added the type/bug The PR fixed a bug or issue reported a bug label Nov 11, 2021
@eolivelli
Copy link
Contributor Author

@congbobo184 PTAL

@sijie sijie mentioned this issue Nov 11, 2021
Closed
@eolivelli
Copy link
Contributor Author

@congbobo184 it may be a regression introduced by #11934

Here I see that a message is being sent,
that's the main difference in the deleteTopic code path in case of force=true and transactions enabled

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

Line 1135 in 8551868

.thenCompose(__ -> transactionBuffer.clearSnapshot()).whenComplete((v, ex) -> {

@eolivelli
Copy link
Contributor Author

@gaoran10 PTAL as well

@codelipenghui codelipenghui modified the milestone: 2.10.0 Nov 11, 2021
@congbobo184 congbobo184 mentioned this issue Nov 11, 2021
Merged
congbobo184 added a commit that referenced this issue Nov 12, 2021
@congbobo184
[Transaction] Fix topicTransactionBuffer handle null snapshot (#12758)
c90c89b 
fix #12754
### Motivation
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

### Modifications
judge NPE logic
eolivelli pushed a commit that referenced this issue Nov 12, 2021
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (#12758)
94db24a 
fix #12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

judge NPE logic

(cherry picked from commit c90c89b)
eolivelli pushed a commit to eolivelli/pulsar that referenced this issue Nov 29, 2021
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (apache…
2258b95 
…#12758)

fix apache#12754
### Motivation
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

### Modifications
judge NPE logic
eolivelli pushed a commit to eolivelli/pulsar that referenced this issue Mar 1, 2022
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (apache…
ba276f3 
…#12758)

fix apache#12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

judge NPE logic

(cherry picked from commit c90c89b)
@eolivelli eolivelli mentioned this issue Mar 1, 2022
Merged
eolivelli pushed a commit to eolivelli/pulsar that referenced this issue Mar 1, 2022
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (apache…
0524a71 
…#12758)

fix apache#12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

judge NPE logic

(cherry picked from commit c90c89b)
eolivelli pushed a commit to datastax/pulsar that referenced this issue Mar 1, 2022
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (apache…
75cec6d 
…#12758)

fix apache#12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

judge NPE logic

(cherry picked from commit c90c89b)
eolivelli pushed a commit to datastax/pulsar that referenced this issue Mar 1, 2022
@congbobo184 @eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (apache…
190e650 
…#12758)

fix apache#12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

judge NPE logic

(cherry picked from commit c90c89b)
(cherry picked from commit 75cec6d)
codelipenghui pushed a commit that referenced this issue Mar 2, 2022
@eolivelli
[Transaction] Fix topicTransactionBuffer handle null snapshot (#12758) (
70e533a 
#14510)

fix #12754
Now when delete topic, we will write a null value to Transaction buffer snapshot topic, other topic recover by this transaction buffer snapshot system topic, will produce NPE

(cherry picked from commit c90c89b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congbobo184 congbobo184

Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Transaction] Fix topicTransactionBuffer handle null snapshot congbobo184/pulsar
3 participants
@eolivelli @codelipenghui @congbobo184