[Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions #14367
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I need more eyes to help me review this change to avoid regression. |
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
Show resolved
Hide resolved
|
Thanks for working on this @mattisonchao! |
|
@mattisonchao - I should have added in my review that your tests look good. |
michaeljmarshall
requested review from
codelipenghui,
aahmed-se,
eolivelli and
Jason918
michaeljmarshall
added
area/broker
type/bug
Technoboy-
requested changes
Feb 18, 2022
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
Show resolved
Hide resolved
mattisonchao
changed the title
|
@michaeljmarshall @Technoboy- |
mattisonchao
changed the title
|
/pulsarbot rerun-failure-checks |
|
@codelipenghui @Technoboy- @eolivelli @lhotari @michaeljmarshall @Jason918 PTAL, when you have time ~ |
LGTM. |
|
I will review this within the next couple hours, thank you for your work @mattisonchao! |
codelipenghui
approved these changes
Mar 4, 2022
|
@codelipenghui if we cut a new rc for 2.10 is would be great to have this patch. |
|
@mattisonchao |
2 tasks
codelipenghui
added
cherry-picked/branch-2.8
2 tasks
gaoran10
pushed a commit
to gaoran10/pulsar
that referenced
this pull request
Mar 11, 2022
…che#14367) (cherry picked from commit 4f1e39b)
gaoran10
pushed a commit
to gaoran10/pulsar
that referenced
this pull request
Mar 11, 2022
…che#14367) (cherry picked from commit 4f1e39b)
codelipenghui
pushed a commit
that referenced
this pull request
Mar 12, 2022
) (cherry picked from commit 4f1e39b)
codelipenghui
pushed a commit
that referenced
this pull request
Mar 14, 2022
…#14658) Original PR #14367 Because the PR #14367 is based on PR #14091, so I want to cherry-pick these two PRs to branch-2.9, the PR #13525 is also needed. --- Fix Issue: #14362 ### Motivation According to relative PR #7724, we will force delete all subscriptions when calling ``healthCheck`` REST API. but it has a race condition when two threads call this API. Please consider this case: > Thread A: Clean up all subscriptions, then create a reader. > Thread B: Prepare to force delete all subscriptions. So, in this case, the reader of thread A is deleted and then an NPE or other exception occurs. ### Modifications - Use ``Completable#handle`` to fix problem 1, the reader needs to be closed regardless of whether there is an exception. - Recheck the subscription after closing reading, and force deletion if it still exists after closing reading. - Added multi-threaded tests for health checks.
|
The #14658 cherry-pick this PR to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master Issue: #14362
Motivation
See #14362
According to relative PR #7724, we will force delete all subscriptions when calling
healthCheckREST API. but it has a race condition when two threads call this API.Please consider this case:
So, in this case, the reader of thread A is deleted and then an NPE or other exception occurs.
Modifications
Completable#handleto fix problem 1, the reader needs to be closed regardless of whether there is an exception.Verifying this change
Documentation
no-need-doc