[broker] Avoid unnecessary recalculation of maxSubscriptionsPerTopic in AbstractTopic #12658
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
doc
| } | ||
|
|
||
| private void updateValue() { | ||
| VALUE_UPDATER.updateAndGet(this, (preValue) -> { |
There was a problem hiding this comment.
We might need to handle the delete policies? Such as the topic level policy being removed, we should use the namespace level policy if present.
There was a problem hiding this comment.
Yes, in this case, updateTopicValue(null) should be called, and the value will be updated to namespaceValue if it's presented.
There was a problem hiding this comment.
Checked with org.apache.pulsar.broker.admin.v2.PersistentTopics#removeMaxSubscriptionsPerTopic, internalSetMaxSubscriptionsPerTopic(null) is called to update the policy.
codelipenghui
added
release/2.8.3
release/2.9.1
type/enhancement
codelipenghui
requested review from
hangc0276,
315157973,
eolivelli and
merlimat
codelipenghui
approved these changes
Nov 8, 2021
zeo1995
pushed a commit
to zeo1995/pulsar
that referenced
this pull request
Nov 9, 2021
* up/master: [Doc] Add explanations for setting geo-replication at topic level (apache#12633) commit chapter Tiered Storage (apache#12592) [pulsar-admin] Add remove-subscription-types-enabled command for namespace (apache#12392) Enable CLI to publish non-batched messages (apache#12641) [Doc] Add doc for tokenSettingPrefix (apache#12662) [pulsar-admin] Add corresponding get command for namespace (apache#12322) [pulsar-admin] Perfect judgment conditions of pulsar-admin (apache#12315) [broker] Avoid unnecessary recalculation of maxSubscriptionsPerTopic in AbstractTopic (apache#12658) [Transaction]Stop TB recovering with exception (apache#12636) [website][upgrade]feat: docs migration - 2.7.1 / client (apache#12612) [website][upgrade]feat: docs migration - 2.7.1 / performance (apache#12611) [website][upgrade]feat: docs migration - 2.7.1 / security (apache#12610) [Modernizer] Apply Modernizer plugin for pulsar broker common module and fix violation. (apache#12657) [Authorization] Support GET_METADATA topic op after enable auth (apache#12656) Fix StringIndexOutOfBoundsException in org.apache.pulsar.broker.resources.NamespaceResources#pathIsFromNamespace (apache#12659)
zeo1995
pushed a commit
to zeo1995/pulsar
that referenced
this pull request
Nov 9, 2021
* up/master: [Doc] Add explanations for setting geo-replication at topic level (apache#12633) commit chapter Tiered Storage (apache#12592) [pulsar-admin] Add remove-subscription-types-enabled command for namespace (apache#12392) Enable CLI to publish non-batched messages (apache#12641) [Doc] Add doc for tokenSettingPrefix (apache#12662) [pulsar-admin] Add corresponding get command for namespace (apache#12322) [pulsar-admin] Perfect judgment conditions of pulsar-admin (apache#12315) [broker] Avoid unnecessary recalculation of maxSubscriptionsPerTopic in AbstractTopic (apache#12658) [Transaction]Stop TB recovering with exception (apache#12636) [website][upgrade]feat: docs migration - 2.7.1 / client (apache#12612) [website][upgrade]feat: docs migration - 2.7.1 / performance (apache#12611) [website][upgrade]feat: docs migration - 2.7.1 / security (apache#12610) [Modernizer] Apply Modernizer plugin for pulsar broker common module and fix violation. (apache#12657) [Authorization] Support GET_METADATA topic op after enable auth (apache#12656) Fix StringIndexOutOfBoundsException in org.apache.pulsar.broker.resources.NamespaceResources#pathIsFromNamespace (apache#12659) # Conflicts: # site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json
2 tasks
eolivelli
pushed a commit
to eolivelli/pulsar
that referenced
this pull request
Nov 29, 2021
…in AbstractTopic (apache#12658) ### Motivation Currently, `AbstractTopic#maxSubscriptionsPerTopic` stores the value of namespace level policy, but we have broker level setting in `org.apache.pulsar.broker.ServiceConfiguration#maxSubscriptionsPerTopic` and topic level setting in `org.apache.pulsar.common.policies.data.TopicPolicies#maxSubscriptionsPerTopic`. And the real value we used of `maxSubscriptionsPerTopic` is calculated every time in `org.apache.pulsar.broker.service.persistent.PersistentTopic#checkMaxSubscriptionsPerTopicExceed`. It can be avoided by cache maxSubscriptionsPerTopic result locally. As we have already registered listeners to namespace policy and topic policy updates, so we can cache the value locally in AbstractTopic to avoid the recalculation. Finally, as some of other policies value have similar issues, I am introducing `PolicyHierarchyValue` to solve the hierarchy value storage and calculation. ### Modifications Introduce `PolicyHierarchyValue` to store policy value in broker, namespace and topic level. It provides corresponding `updateXX` methods, and recalculate real value in it. And the `get()` method returns the policy value we should use directly.
23 tasks
23 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently,
AbstractTopic#maxSubscriptionsPerTopicstores the value of namespace level policy, but we have broker level setting inorg.apache.pulsar.broker.ServiceConfiguration#maxSubscriptionsPerTopicand topic level setting inorg.apache.pulsar.common.policies.data.TopicPolicies#maxSubscriptionsPerTopic.And the real value we used of
maxSubscriptionsPerTopicis calculated every time inorg.apache.pulsar.broker.service.persistent.PersistentTopic#checkMaxSubscriptionsPerTopicExceed. It can be avoided by cache maxSubscriptionsPerTopic result locally.As we have already registered listeners to namespace policy and topic policy updates, so we can cache the value locally in AbstractTopic to avoid the recalculation.
Finally, as some of other policies value have similar issues, I am introducing
PolicyHierarchyValueto solve the hierarchy value storage and calculation.Modifications
Introduce
PolicyHierarchyValueto store policy value in broker, namespace and topic level.It provides corresponding
updateXXmethods, and recalculate real value in it. And theget()method returns the policy value we should use directly.Verifying this change
This change is already covered by existing tests, such as
org.apache.pulsar.broker.admin.TopicPoliciesTest#testMaxSubscriptionsPerTopicAnd added PolicyHierarchyValueTest for new class.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
docNew class PolicyHierarchyValue comes with docs.
The rest is internal optimization, no user behavior changed.