Add bundle split capabilities in ModularLoadManagerImpl #385
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdhabalia
added
type/enhancement
merlimat
reviewed
May 12, 2017
| @@ -289,28 +290,38 @@ public Boolean call() throws Exception { | |||
|
|
|||
| // Broker doesn't restart on global-zk session lost: so handling unexpected exception | |||
| try { | |||
| this.zkSession.get().getData(path, watcher, (rc, path1, ctx, content, stat) -> { | |||
| // Put watch on the node no matter what | |||
There was a problem hiding this comment.
Can you describe this change a bit more? Since it doesn't affect only the bundle splitting in the load manager, it would be better to split it in a separate PR.
There was a problem hiding this comment.
Sure! This uses the capability of ZooKeeper.exists to put a watch on nonexistent nodes. The absence of this behavior caused an issue in which namespace bundle boundary updates were not being seen across all brokers. I will make a separate PR w/ just the cache change.
|
should we close this PR in favor of #857 ? |
|
Yes closing for #857 |
hrsakai
pushed a commit
to hrsakai/pulsar
that referenced
this pull request
Dec 10, 2020
…#385) Signed-off-by: Pavel Agaletskiy <psagaletskiy@avito.ru> Co-authored-by: Pavel Agaletskiy <psagaletskiy@avito.ru>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
ModularLoadManagerImpllacked bundle split capabilities. Additionally, automatic bundle splits reassigned the bundles to the same broker when the bundle could more usefully be reassigned after being split.Modifications
Roughly in order of importance:
ModularLoadManagerImpl.ZooKeeperCachewill now always put a watch on a node, even if it did not previously exist. This is done via using the capability ofZooKeeper.existsto put a watch on non-existent nodes.BundleSplitStrategyhas been added to allow the injection of automatic bundle splitting behavior.loadBalancerAutoBundleSplitEnabledis now a dynamic configuration.BundleData->TimeAverageBundleDataBundleDataholds non-time average (numBundles/Topics, etc) bundle valuesModularLoadManagerImplsuch asModularLoadManagerStrategyare now exposed thePulsarServiceinstance instead ofServiceConfigurationto give more information and freedom.Result
This PR adds a bundle split mechanism for
ModularLoadManagerImpland changes bundle splitting behavior so that bundles which are split are also unloaded, so that the splits may be assigned to new brokers.