[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18283

codelipenghui · 2022-11-01T13:14:34Z

Motivation

The current behavior for the delete_when_subscriptions_caught_up strategy is not expected. The active consumer will not be closed even if users enable delete_when_subscriptions_caught_up and there are no backlogs for the topic.

It should be the part that #6077 has missed. And Sijie has mentioned in the comment #6077 (review).

To correct the behavior of delete_when_subscriptions_caught_up

Modifications

Close active consumers if delete_when_subscriptions_caught_up is applied and there are no backlogs for the topic. So that the topic can be cleaned up properly by the topic GC thread.

Verifying this change

Updated the existing test.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: codelipenghui#17

…ile have active consumers

codecov-commenter · 2022-11-02T04:16:49Z

Codecov Report

Attention: Patch coverage is 52.38095% with 30 lines in your changes missing coverage. Please review.

Project coverage is 40.13%. Comparing base (0866c3a) to head (f8169c9).
Report is 2237 commits behind head on master.

Files with missing lines	Patch %	Lines
...sar/broker/service/persistent/PersistentTopic.java	52.38%	27 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18283      +/-   ##
============================================
+ Coverage     38.97%   40.13%   +1.15%     
- Complexity     8311     8651     +340     
============================================
  Files           683      687       +4     
  Lines         67325    67437     +112     
  Branches       7217     7225       +8     
============================================
+ Hits          26239    27063     +824     
+ Misses        38079    37349     -730     
- Partials       3007     3025      +18

Flag	Coverage Δ
unittests	`40.13% <52.38%> (+1.15%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...sar/broker/service/persistent/PersistentTopic.java	`60.90% <52.38%> (+3.92%)`	⬆️

... and 71 files with indirect coverage changes

Technoboy- · 2022-11-02T04:25:58Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

-                    //  2. We want to kick out everyone and forcefully delete the topic.
-                    //     In this case, we shouldn't care if the usageCount is 0 or not, just proceed
-                    if (currentUsageCount() ==  0 || (closeIfClientsConnected && !failIfHasSubscriptions)) {
+                    if (currentUsageCount() == 0) {


Seems we need to keep the original condition (closeIfClientsConnected && !failIfHasSubscriptions))

I feel the current code is correct. Because we have called disconnection operation before. If there are still any connections at the current time, it means that a new connection has come in, we should consider that there is a concurrency problem and give up deleting the topic.

Yes, you are right @poorbarcode

The closeIfClientsConnected and failIfHasSubscriptions is validated before we reach here. It looks like duplicated validation.

poorbarcode · 2022-11-02T05:38:43Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

-                    //  2. We want to kick out everyone and forcefully delete the topic.
-                    //     In this case, we shouldn't care if the usageCount is 0 or not, just proceed
-                    if (currentUsageCount() ==  0 || (closeIfClientsConnected && !failIfHasSubscriptions)) {
+                    if (currentUsageCount() == 0) {


I feel the current code is correct. Because we have called disconnection operation before. If there are still any connections at the current time, it means that a new connection has come in, we should consider that there is a concurrency problem and give up deleting the topic.

315157973 · 2022-11-02T06:08:52Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

+            } else if (!closeIfClientsConnected && currentUsageCount() != 0 && !failIfHasBacklogs) {
+                return FutureUtil.failedFuture(new TopicBusyException(
+                        "Topic has " + currentUsageCount() + " connected producers/consumers"));


It needs to fail as long as there are connections, so it has nothing to do with !failIfHasBacklogs.

Or if failIfHasBacklogs is triggered, the error message returned should not be like this

Hi @315157973

When failIfHasBacklogs is true, the expected behavior is that:

failure of any producer exists

if any consumer exists, close connections

Actually, the code should be like this:

if ( !closeIfClientsConnected && currentUsageCount() ){ fail... } else if ( !closeIfClientsConnected !failIfHasBacklogs && anyProducerExists() ){ fail... }

This code may not be easy to understand from the context, but the logic is correct

There is also a concurrency problem that may lead to the incorrect deletion of existing producer topics. E.g:

checkGC new producer registry

ensure no producer exists

delete topic ( false, true, false)

ensure no backlog

new producer registry

fence topic

disconnect all clients

delete topic

The above flow shows the case: producer exists but the topic is deleted. Would it be better to add a double-check？

@poorbarcode We have write lock here https://github.com/apache/pulsar/pull/18283/files#diff-5edf14cc6f25857d0cfdd26b2d3b3141230ecfb0dfa95aebf7583fd76ede4c4bR1161

@codelipenghui

We have write lock here https://github.com/apache/pulsar/pull/18283/files#diff-5edf14cc6f25857d0cfdd26b2d3b3141230ecfb0dfa95aebf7583fd76ede4c4bR1161

Ah, yes you are right

mattisonchao

LGTM

congbobo184 · 2022-11-15T02:09:06Z

could you please cherry-pick this PR to branch-2.9? thanks.

codelipenghui · 2022-11-16T08:25:51Z

#18320 has cherry-picked this one to branch-2.11

codelipenghui · 2022-11-16T12:29:31Z

f5c9354 cherry-picked to branch-2.10

codelipenghui · 2022-11-16T12:44:48Z

3cf167a cherry-picked to branch-2.9

…ile have active consumers (#18283) (cherry picked from commit 67d9d63)

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work wh…

d230b62

…ile have active consumers

codelipenghui added type/bug The PR fixed a bug or issue reported a bug area/broker release/2.9.4 release/2.11.1 release/2.10.3 labels Nov 1, 2022

codelipenghui added this to the 2.12.0 milestone Nov 1, 2022

codelipenghui self-assigned this Nov 1, 2022

codelipenghui requested review from eolivelli, Jason918, 315157973, mattisonchao and Technoboy- November 1, 2022 13:15

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 1, 2022

Technoboy- added the ready-to-test label Nov 2, 2022

Technoboy- closed this Nov 2, 2022

Technoboy- reopened this Nov 2, 2022

Technoboy- reviewed Nov 2, 2022

View reviewed changes

poorbarcode approved these changes Nov 2, 2022

View reviewed changes

315157973 reviewed Nov 2, 2022

View reviewed changes

codelipenghui and others added 3 commits November 2, 2022 15:55

Fix comment.

b67eca2

Remove meaningless check.

f4e3f28

fix compile issue.

f8169c9

Technoboy- approved these changes Nov 2, 2022

View reviewed changes

mattisonchao approved these changes Nov 3, 2022

View reviewed changes

poorbarcode approved these changes Nov 3, 2022

View reviewed changes

coderzc approved these changes Nov 3, 2022

View reviewed changes

315157973 approved these changes Nov 3, 2022

View reviewed changes

codelipenghui merged commit 67d9d63 into apache:master Nov 3, 2022

Technoboy- mentioned this pull request Nov 7, 2022

[improve][test] Add subscribing regex topic test for delete_when_subscriptions_caught_up. #18368

Merged

4 tasks

codelipenghui added the cherry-picked/branch-2.11 label Nov 16, 2022

codelipenghui added the cherry-picked/branch-2.10 label Nov 16, 2022

codelipenghui added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Nov 16, 2022

mattisonchao mentioned this pull request Nov 21, 2022

[fix][schema] Fix cherry-pick issue from #18283 #18555

Merged

5 tasks

mattisonchao added a commit that referenced this pull request Nov 21, 2022

[fix][schema] Fix cherry-pick issue from #18283 (#18555)

9343a35

mattisonchao added a commit that referenced this pull request Nov 21, 2022

[fix][schema] Fix cherry-pick issue from #18283 (#18555)

0896eda

congbobo184 pushed a commit that referenced this pull request Nov 27, 2022

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work wh…

17f23df

…ile have active consumers (#18283) (cherry picked from commit 67d9d63)

codelipenghui deleted the penghui/fix-subscription-catchup-deletion branch March 26, 2024 00:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18283

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18283

codelipenghui commented Nov 1, 2022

codecov-commenter commented Nov 2, 2022 •

edited

Loading

Technoboy- Nov 2, 2022

poorbarcode Nov 2, 2022

codelipenghui Nov 2, 2022

poorbarcode Nov 2, 2022

315157973 Nov 2, 2022

poorbarcode Nov 2, 2022 •

edited

Loading

poorbarcode Nov 2, 2022

codelipenghui Nov 2, 2022

poorbarcode Nov 2, 2022 •

edited

Loading

mattisonchao left a comment

congbobo184 commented Nov 15, 2022

codelipenghui commented Nov 16, 2022 •

edited

Loading

codelipenghui commented Nov 16, 2022

codelipenghui commented Nov 16, 2022

`checkGC`	`new producer registry`
ensure no producer exists
delete topic ( `false`, `true`, `false`)
ensure no backlog
	new producer registry
fence topic
disconnect all clients
delete topic

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18283

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers #18283

Conversation

codelipenghui commented Nov 1, 2022

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

codecov-commenter commented Nov 2, 2022 • edited Loading

Codecov Report

Technoboy- Nov 2, 2022

Choose a reason for hiding this comment

poorbarcode Nov 2, 2022

Choose a reason for hiding this comment

codelipenghui Nov 2, 2022

Choose a reason for hiding this comment

poorbarcode Nov 2, 2022

Choose a reason for hiding this comment

315157973 Nov 2, 2022

Choose a reason for hiding this comment

poorbarcode Nov 2, 2022 • edited Loading

Choose a reason for hiding this comment

poorbarcode Nov 2, 2022

Choose a reason for hiding this comment

codelipenghui Nov 2, 2022

Choose a reason for hiding this comment

poorbarcode Nov 2, 2022 • edited Loading

Choose a reason for hiding this comment

mattisonchao left a comment

Choose a reason for hiding this comment

congbobo184 commented Nov 15, 2022

codelipenghui commented Nov 16, 2022 • edited Loading

codelipenghui commented Nov 16, 2022

codelipenghui commented Nov 16, 2022

codecov-commenter commented Nov 2, 2022 •

edited

Loading

poorbarcode Nov 2, 2022 •

edited

Loading

poorbarcode Nov 2, 2022 •

edited

Loading

codelipenghui commented Nov 16, 2022 •

edited

Loading