Disable stats recorder for built-in PulsarClient #12217
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BewareMyPower
added
area/broker
doc-not-needed
BewareMyPower
requested review from
eolivelli,
hangc0276,
codelipenghui,
315157973,
aahmed-se,
merlimat and
sijie
and removed request for
eolivelli
hangc0276
reviewed
Sep 28, 2021
michaeljmarshall
approved these changes
Sep 28, 2021
There was a problem hiding this comment.
LGTM.
If we find that we are interested in these stats, I think it makes more sense to update the implementation to add them to the results returned by the metrics endpoint instead of hide the stats in logs. Technically, this could be useful for client applications as well. The pulsar go client creates prometheus metrics, and I've found those very useful.
hangc0276
approved these changes
Sep 28, 2021
codelipenghui
approved these changes
Sep 28, 2021
315157973
approved these changes
Sep 28, 2021
|
My only concern is that some users conduct some business analysis by collecting this log. At least this change needs to be explained in the Release, right? |
Yeah, you're right. Anyway it's a trade-off. And I'm curious about how can we explain it in release notes? |
merlimat
approved these changes
Sep 28, 2021
codelipenghui
pushed a commit
that referenced
this pull request
Sep 29, 2021
### Motivation In `PulsarService`, there's a built-in client used in many places, like topic compaction, system topics read/write. It could also be used in protocol handlers. Different with the manually created Pulsar client, it can read the configured authentication params and TLS related configs for broker-to-broker communication. However, it doesn't change the default `statsIntervalSeconds` config, which means each time a producer/consumer/reader is created, a stats recorder (`ProducerStatsRecorderImpl` or `ConsumerStatsRecorderImpl`) will be created, in which there's a Netty `TimerTask` that prints stats periodically (the default interval is 1 minute). If there're many producers/consumers/readers created by this built-in client, the unnecessary CPU usage will be high. ### Modifications Configure the `statsIntervalSeconds` with zero value to disable stats recorder. (cherry picked from commit 52232e6)
eolivelli
pushed a commit
to datastax/pulsar
that referenced
this pull request
Nov 23, 2021
### Motivation In `PulsarService`, there's a built-in client used in many places, like topic compaction, system topics read/write. It could also be used in protocol handlers. Different with the manually created Pulsar client, it can read the configured authentication params and TLS related configs for broker-to-broker communication. However, it doesn't change the default `statsIntervalSeconds` config, which means each time a producer/consumer/reader is created, a stats recorder (`ProducerStatsRecorderImpl` or `ConsumerStatsRecorderImpl`) will be created, in which there's a Netty `TimerTask` that prints stats periodically (the default interval is 1 minute). If there're many producers/consumers/readers created by this built-in client, the unnecessary CPU usage will be high. ### Modifications Configure the `statsIntervalSeconds` with zero value to disable stats recorder. (cherry picked from commit 52232e6)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
### Motivation In `PulsarService`, there's a built-in client used in many places, like topic compaction, system topics read/write. It could also be used in protocol handlers. Different with the manually created Pulsar client, it can read the configured authentication params and TLS related configs for broker-to-broker communication. However, it doesn't change the default `statsIntervalSeconds` config, which means each time a producer/consumer/reader is created, a stats recorder (`ProducerStatsRecorderImpl` or `ConsumerStatsRecorderImpl`) will be created, in which there's a Netty `TimerTask` that prints stats periodically (the default interval is 1 minute). If there're many producers/consumers/readers created by this built-in client, the unnecessary CPU usage will be high. ### Modifications Configure the `statsIntervalSeconds` with zero value to disable stats recorder.
congbobo184
added
cherry-picked/branch-2.9
congbobo184
pushed a commit
that referenced
this pull request
Nov 10, 2022
### Motivation In `PulsarService`, there's a built-in client used in many places, like topic compaction, system topics read/write. It could also be used in protocol handlers. Different with the manually created Pulsar client, it can read the configured authentication params and TLS related configs for broker-to-broker communication. However, it doesn't change the default `statsIntervalSeconds` config, which means each time a producer/consumer/reader is created, a stats recorder (`ProducerStatsRecorderImpl` or `ConsumerStatsRecorderImpl`) will be created, in which there's a Netty `TimerTask` that prints stats periodically (the default interval is 1 minute). If there're many producers/consumers/readers created by this built-in client, the unnecessary CPU usage will be high. ### Modifications Configure the `statsIntervalSeconds` with zero value to disable stats recorder. (cherry picked from commit 52232e6)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
In
PulsarService, there's a built-in client used in many places, like topic compaction, system topics read/write. It could also be used in protocol handlers. Different with the manually created Pulsar client, it can read the configured authentication params and TLS related configs for broker-to-broker communication.However, it doesn't change the default
statsIntervalSecondsconfig, which means each time a producer/consumer/reader is created, a stats recorder (ProducerStatsRecorderImplorConsumerStatsRecorderImpl) will be created, in which there's a NettyTimerTaskthat prints stats periodically (the default interval is 1 minute). If there're many producers/consumers/readers created by this built-in client, the unnecessary CPU usage will be high.Modifications
Configure the
statsIntervalSecondswith zero value to disable stats recorder.Verifying this change
This change is a trivial rework / code cleanup without any test coverage.