Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed race condition on deleting topic with active readers #7715

Merged
merged 1 commit into from
Aug 2, 2020

Conversation

merlimat
Copy link
Contributor

@merlimat merlimat commented Aug 1, 2020

Motivation

When forcifully deleting a topic that has active readers, there is a race condition that lead to a failure in deleting the topic.
The race is

  1. Connected readers are closed and removed
  2. Since there's no active consumer on a non-durable subscription (eg: the reader sub), then we trigger the removal of the subscription and its cursor. That happens in background, in order to avoid deadlocking. See
    topic.getBrokerService().pulsar().getExecutor().submit(() ->{
  3. Once all consumers are closed, we then proceed in deleting the managed-ledger
  4. To delete the managed-ledger, we get the current list of cursors and delete all of them in parallel
  5. Since the cursors are removed in background, the list that we got might contain cursors that don't exist anymore when we try to delete them.
org.apache.bookkeeper.mledger.ManagedLedgerException: ManagedCursor not found: reader-7d300cd535
	at org.apache.pulsar.broker.service.persistent.PersistentTopic$4.deleteLedgerFailed(PersistentTopic.java:884) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$16.deleteCursorFailed(ManagedLedgerImpl.java:2177) ~[org.apache.pulsar-managed-ledger-2.6.0.jar:2.6.0]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncDeleteCursor(ManagedLedgerImpl.java:766) ~[org.apache.pulsar-managed-ledger-2.6.0.jar:2.6.0]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncDelete(ManagedLedgerImpl.java:2156) ~[org.apache.pulsar-managed-ledger-2.6.0.jar:2.6.0]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$null$19(PersistentTopic.java:859) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) ~[?:?]
	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$delete$20(PersistentTopic.java:853) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
	at java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.delete(PersistentTopic.java:843) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.deleteForcefully(PersistentTopic.java:791) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
	at org.apache.pulsar.broker.service.BrokerService.deleteTopic(BrokerService.java:719) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: ManagedCursor not found: reader-7d300cd535

On topic deletion, we should be more tolerant of disappearing cursors and instead proceed as a success case.

@merlimat merlimat added the type/bug The PR fixed a bug or issue reported a bug label Aug 1, 2020
@merlimat merlimat added this to the 2.7.0 milestone Aug 1, 2020
@merlimat merlimat self-assigned this Aug 1, 2020
@srkukarni srkukarni merged commit 03131fb into apache:master Aug 2, 2020
@merlimat merlimat deleted the cursor-delete-race branch August 5, 2020 22:26
jerrypeng pushed a commit to jerrypeng/incubator-pulsar that referenced this pull request Aug 14, 2020
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Sep 5, 2020
lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Sep 5, 2020
lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Sep 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants