[pulsar-client] Process partitioned-topic messages on different listener-threads #10017
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdhabalia
requested review from
merlimat,
sijie,
jiazhai,
codelipenghui and
nkurihar
eolivelli
requested changes
Mar 23, 2021
| @@ -32,11 +32,15 @@ | |||
|
|
|||
| private final Message<T> msg; | |||
| private final TopicMessageIdImpl messageId; | |||
| // consumer if this message is received by that consumer | |||
| ConsumerImpl receivedByconsumer; | |||
| this.topicPartitionName = topicPartitionName; | ||
| this.receivedByconsumer = receivedByConsumer; |
There was a problem hiding this comment.
can we add an assertion that receivedByConsumer is non null ?
There was a problem hiding this comment.
receivedByConsumer can be null if you want to create a message without consumer for some other purpose. so, it's intentional.
|
/pulsarbot run-failure-checks |
codelipenghui
approved these changes
Mar 24, 2021
michaeljmarshall
added
cherry-picked/branch-2.7
|
Cherry picking this to branch-2.7 because #11455 relies on part of it, and it simplifies the conflict resolution for the other commit. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Right now, Consumer of partitioned topic always uses only 1 listener thread to call listener on received message even though pulsar-client is created with multiple listener-theads. So, if user creates Pulsar-client with 10 listener threads and creates consumer for partitoned-topic with 10 partitions then consumer will use only 1 listener threads to serve all 10 partitions instead utilizing all 10 threads. This creates a bottleneck if one of the partition is slow, CPU spins on only one thread and there is no way to make overall process faster.
Therefore, consumer should use each partition's listener-thread to process message received by that partition. This helps:
Modification
allow listener-processing of each partition on its dedicated listener-thread instead on common partitioned-topic's listener thread.