[improve][broker] Do not close the socket if lookup failed due to LockBusyException #21993
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…kBusyException
### Motivation
When a broker restarted, there is a case in
`NamespaceService#findBrokerServiceUrl`:
1. `ownershipCache.getOwnerAsync(bundle)` got an empty data, then
`searchForCandidateBroker` will be called
2. The broker itself was elected as the candidate broker.
3. Meanwhile, the other broker has acquired the distributed lock of the
bundle, then `ownershipCache.tryAcquiringOwnership` will fail with
```java
lookupFuture.completeExceptionally(new PulsarServerException(
"Failed to acquire ownership for namespace bundle " + bundle, exception));
```
See apache/pulsar-client-cpp#390 for the real
world case.
Then in `TopicLookupBase#handleLookupError`, this exception will be
wrapped into a `ServiceNotReady` error to client.
This case happens very frequently in our production environment when a
broker restarted. If there is a `PulsarClient` that has many producers
or consumers, the connection will be closed, which results in many
reconnections, which brings much pressure to the cluster.
### Modifications
In `handleLookupError`, check the `PulsarServerException` and unwrap the
`CompletionException`. If the unwrapped exception is
`MetadataStoreException`, return the `MetadataError` to avoid closing
the connection at client side.
Add `testLookupConnectionNotCloseIfFailedToAcquireOwnershipOfBundle` to
simulate the case and verify the socket won't be closed.
BewareMyPower
added
type/enhancement
BewareMyPower
requested review from
Technoboy-,
315157973,
codelipenghui,
Demogorgon314,
poorbarcode and
heesung-sn
|
@BewareMyPower Looks like a good proposal, I'll review later in more detail. One observation is that this problem is really common when the leader broker election is broken, like with the bug #21897 which was fixed by #21894 . |
poorbarcode
approved these changes
Jan 31, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #21993 +/- ##
============================================
- Coverage 73.63% 73.63% -0.01%
- Complexity 32473 32494 +21
============================================
Files 1863 1863
Lines 138786 138814 +28
Branches 15207 15216 +9
============================================
+ Hits 102201 102216 +15
- Misses 28687 28695 +8
- Partials 7898 7903 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Technoboy-
approved these changes
Jan 31, 2024
Demogorgon314
approved these changes
Jan 31, 2024
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/TopicLookupBase.java
Show resolved
Hide resolved
pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/TopicLookupBase.java
Show resolved
Hide resolved
Technoboy-
pushed a commit
that referenced
this pull request
Feb 5, 2024
Technoboy-
pushed a commit
that referenced
this pull request
Feb 27, 2024
mukesh-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 1, 2024
…kBusyException (apache#21993) (cherry picked from commit bf5639f)
mukesh-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Mar 6, 2024
…kBusyException (apache#21993) (cherry picked from commit bf5639f)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When a broker restarted, there is a case in
NamespaceService#findBrokerServiceUrl:ownershipCache.getOwnerAsync(bundle)got an empty data, thensearchForCandidateBrokerwill be calledownershipCache.tryAcquiringOwnershipwill fail withSee apache/pulsar-client-cpp#390 for the real world case.
Then in
TopicLookupBase#handleLookupError, this exception will be wrapped into aServiceNotReadyerror to client.This case happens very frequently in our production environment when a broker restarted. If there is a
PulsarClientthat has many producers or consumers, the connection will be closed, which results in many reconnections, which brings much pressure to the cluster.Modifications
In
handleLookupError, check thePulsarServerExceptionand unwrap theCompletionException. If the unwrapped exception isMetadataStoreException, return theMetadataErrorto avoid closing the connection at client side.Add
testLookupConnectionNotCloseIfFailedToAcquireOwnershipOfBundleto simulate the case and verify the socket won't be closed.Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: