-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Issue #7556][pulsar-client] Ensure parallel invocations of MultiTopicsConsumerImpl::subscribeAsync with same topic name do not produce an error. #7691
Merged
sijie
merged 1 commit into
apache:master
from
sandrzejczak:multitopicsconsumerimpl-parallel-subscribeasync
Aug 6, 2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…c with same topic name do not hang. Fixes apache#7556.
sijie
approved these changes
Jul 29, 2020
/pulsarbot run-failure-checks |
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
…c with same topic name do not hang. Fixes apache#7556. (apache#7691) Fixes apache#7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: apache#7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class.
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…c with same topic name do not hang. Fixes apache#7556. (apache#7691) Fixes apache#7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: apache#7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class.
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…c with same topic name do not hang. Fixes apache#7556. (apache#7691) Fixes apache#7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: apache#7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class.
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…c with same topic name do not hang. Fixes apache#7556. (apache#7691) Fixes apache#7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: apache#7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class.
wolfstudy
pushed a commit
that referenced
this pull request
Oct 30, 2020
…c with same topic name do not hang. Fixes #7556. (#7691) Fixes #7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: #7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class. (cherry picked from commit c54a47e)
merlimat
pushed a commit
to merlimat/pulsar
that referenced
this pull request
Dec 19, 2020
…c with same topic name do not hang. Fixes apache#7556. (apache#7691) Fixes apache#7556 ### Motivation We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case. This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync. However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the `topics` map does not contain any entry for the topic yet. More detailed description is available at issue's page: apache#7556 ### Modifications Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if `topics` map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequent `checkState(currentAllTopicsPartitionsNumber == numTopics, "...")` invocation from throwing an exception which would cause topic's consumers to get closed. ### Verifying this change Added new unit test to MultiTopicsConsumerImplTest class.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
release/2.6.2
type/enhancement
The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #7556
Motivation
We use PatternMultiTopicsConsumerImpl in our solution, but we need to subscribe to new topics instantly after their creation. Default behavior of PatternMultiTopicsConsumerImpl is too slow and too costly for our use case.
This is why we've created an external mechanism which notifies us about new topics. We then subscribe to them using MultiTopicsConsumerImpl#subscribeAsync.
However, if this method is invoked multiple times with same topicName simultaneously, an error may occur and topic's consumers may get closed. It happens, because multiple invocations can pass the initial check in topicNameValid(String topicName), as the
topics
map does not contain any entry for the topic yet. More detailed description is available at issue's page: #7556Modifications
Code in MultiTopicsConsumerImpl#doSubscribeTopicPartitions now checks if
topics
map already contains an entry for a given topic. It does so by checking the return value of putIfAbsent method. If it does already contain an entry, then subscribeResult future is completed exceptionally and the method returns. It prevents subsequentcheckState(currentAllTopicsPartitionsNumber == numTopics, "...")
invocation from throwing an exception which would cause topic's consumers to get closed.Verifying this change
Added new unit test to MultiTopicsConsumerImplTest class.
Does this pull request potentially affect one of the following parts:
Documentation