[feature][broker]Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

HQebupt · 2022-06-29T09:57:17Z

email discussion thread: https://lists.apache.org/thread/36vyyhndr4og175k2bz3mfdf5ctd2xky

Motivation

See PIP-182 #16274

Modifications

The main idea of the new strategy is to unify the requirement of load shedding strategy and bundle placement strategy, which consider the resource usage with weight, including historical observations.

How to calculate a score for a broker ?

use its historical load and short-term load data with weight, which can solve the case of load jitter in a broker.

How to select a broker for assignning bundle ?

select a broker based on which one has the least average resource usage with weight.
the random selection algorithm is better than the minScore among low load brokers, and use loadBalancerAverageResourceUsageDifferenceThresholdShedderPercentage to adjust the size of the randomization pool

Below are screenshots of the effect of the new strategy with less time and fewer load balancing times.

Verifying this change

Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

doc
Doc added in admin client api.

…tation based on the least resource usage with weight

HQebupt · 2022-06-30T06:03:12Z

/pulsarbot run-failure-checks

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java

conf/broker.conf

...roker/src/test/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategyTest.java

… whether to be a best candidate in LeastResourceUsageWithWeight

heesung-sn · 2022-07-01T17:38:52Z

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java

+        final double diffThreshold =
+                conf.getLoadBalancerAverageResourceUsageDifferenceThresholdShedderPercentage() / 100.0;
+        brokerAvgResourceUsageWithWeight.forEach((broker, avgResUsage) -> {
+            if (avgResUsage + diffThreshold <= avgUsage) {


I think this algo is better than the minScore, but we still have the small randomization pool issue here.
example) b1:4, b2:20, b3:20

Here, still, b1 will be repeatedly selected until b1's load is updated to the leader, though all of these brokers should be candidates as their loads are low. (As we discussed, when many bundles are unloading in a short period, this might lead to b1 ending up overloaded, like b1:90%, b2:20, b3:20)

You are right. The original intention of this algorithm is to select a larger randomization pool. It doesn't work well for small clusters. But there is a compromise improvement point, that is, adjusting the diffThreshold to a negative number, which can expand the randomization pool.
(eg: b1:4, b2:20, b3:20, set diffThreshold=-15, all of brokers can be candidates.)
This diffThreshold parameter takes effect dynamically, so it can be adjusted dynamically according to the status of cluster. Do you have a better suggestion? :)

@heesung-sn @dragonls PTAL

Please explain the negative value in the doc, In the config definition.

Another edge case is that with the negative threshold, it might be selecting overloaded brokers.
example) threshold =-15, {b1=25, b2=100, b3=100, b4=100, b5=100} => b2, b3, b4, b5 will be selected.

And another edge case is that with a low variance of the usage distribution, it might be selecting none of the brokers.
example) threshold =10, {b1=24, b2=25, b3=26} => none will be selected.

If we want to care more about the bigger randomization pool, one could just randomly select any brokers below LoadBalancerBrokerOverloadedThresholdPercentage, which could be just another strategy.

Truely. An algorithm that can be applied to the many cases is good, but cannot cover all cases.

Another edge case is that with the negative threshold, it might be selecting overloaded brokers.
example) threshold =-15, {b1=25, b2=100, b3=100, b4=100, b5=100} => b2, b3, b4, b5 will be selected.

The operator need to know the status of the cluster and configure the threshold appropriately.

negative threshold applies when a large number of brokers have less load.

positive threshold applies when a large number of brokers have heavy load.

And another edge case is that with a low variance of the usage distribution, it might be selecting none of the brokers.
example) threshold =10, {b1=24, b2=25, b3=26} => none will be selected.

Assign randomly if none is selected.

It is good we have a fallback case that just randomly selects any broker, when empty.

Please update the doc to further explain the edge cases.

I prefer just keep the algorithm simple, otherwise it is hard to debug and understand.

The diffThreshold might filter all candidates out and finally just randomly select the broker from all candidates, which might be high load.

The principle of selection is to let the broker with lower load be selected, such as selecting a broker which is under avg load.

It would be great if we could explain more about the negative and positive threshold considerations in the config broker.conf.

Otherwise, LGTM.

eolivelli

I am not a expert of this part.
So I won't say much about the algorithm.

A part from that I believe that the change is good.
we are adding a new pluggable part of Pulsar, so a little discussion on dev@pulsar.apache.org is deserved

thanks for contributing this enhancement

...ar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategy.java

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java

HQebupt · 2022-07-05T11:35:34Z

I am not a expert of this part. So I won't say much about the algorithm.

A part from that I believe that the change is good. we are adding a new pluggable part of Pulsar, so a little discussion on dev@pulsar.apache.org is deserved

thanks for contributing this enhancement

Thx, here is the email discussion thread: https://lists.apache.org/thread/36vyyhndr4og175k2bz3mfdf5ctd2xky. I added it into the PR description.

...roker/src/test/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategyTest.java

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java

HQebupt · 2022-07-06T02:22:45Z

/pulsarbot run-failure-checks

HQebupt · 2022-07-06T10:21:41Z

/pulsarbot run-failure-checks

...ar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategy.java

eolivelli

LGTM

HQebupt · 2022-07-06T14:41:40Z

/pulsarbot run-failure-checks

HQebupt · 2022-07-07T01:50:15Z

@Demogorgon314 Could you please review the PR again ?

conf/broker.conf

HQebupt · 2022-07-08T02:05:50Z

/pulsarbot run-failure-checks

Jason918

LGTM

HQebupt · 2022-07-08T06:18:59Z

/pulsarbot run-failure-checks

HQebupt · 2022-07-08T12:50:48Z

/pulsarbot run-failure-checks

heesung-sn · 2022-07-11T15:27:47Z

LGTM

…tation based on the least resource usage with weight (apache#16281) email discussion thread: https://lists.apache.org/thread/36vyyhndr4og175k2bz3mfdf5ctd2xky ### Motivation See PIP-182 apache#16274 ### Modifications See apache#16274 The main idea of the new strategy is to unify the requirement of load shedding strategy and bundle placement strategy, which consider the resource usage with weight, including historical observations. How to calculate a score for a broker ? - use its historical load and short-term load data with weight, which can solve the case of load jitter in a broker. How to select a broker for assignning bundle ? - select a broker based on which one has the least average resource usage with weight. - the random selection algorithm is better than the `minScore` among low load brokers, and use `loadBalancerAverageResourceUsageDifferenceThresholdShedderPercentage ` to adjust the size of the `randomization pool`

…tation based on the least resource usage with weight (apache#16281)

…ge request !151) Squash merge branch 'release-2.8.1.4-bundle-place' into 'release-2.8.1.4' Fixes #<xyz> ### Motivation --story=878626611 新增基于资源权重的放置策略apache#16281 (merge request !150) TAPD: --story=878626611

[feature][broker]Provide new load balance placement strategy implemen…

28788e0

…tation based on the least resource usage with weight

github-actions bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Jun 29, 2022

Technoboy- assigned HQebupt Jun 29, 2022

add unit test and doc

c12a507

heesung-sn reviewed Jun 30, 2022

View reviewed changes

...roker/src/test/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategyTest.java Show resolved Hide resolved

Jason918 mentioned this pull request Jul 1, 2022

[Improve][broker] When calculating resource usage, use getMaxResourceUsageWithWeight instead of getMaxResourceUsage #16319

Open

4 tasks

HQebupt added 2 commits July 1, 2022 15:49

add more unit test case

b30c205

add average resource usage difference threshold to determine a broker…

6a63c86

… whether to be a best candidate in LeastResourceUsageWithWeight

HQebupt force-pushed the leastResourceUsageWithWeight branch from ce402a8 to 6a63c86 Compare July 1, 2022 10:56

heesung-sn reviewed Jul 1, 2022

View reviewed changes

eolivelli requested changes Jul 5, 2022

View reviewed changes

...ar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategy.java Outdated Show resolved Hide resolved

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java Outdated Show resolved Hide resolved

Demogorgon314 reviewed Jul 5, 2022

View reviewed changes

...roker/src/test/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategyTest.java Outdated Show resolved Hide resolved

...er/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java Outdated Show resolved Hide resolved

make reflection method common

9fe58a4

HQebupt force-pushed the leastResourceUsageWithWeight branch from 449acb4 to 9fe58a4 Compare July 5, 2022 11:40

HQebupt requested a review from eolivelli July 6, 2022 02:22

eolivelli requested changes Jul 6, 2022

View reviewed changes

...ar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/ModularLoadManagerStrategy.java Show resolved Hide resolved

fast fail if LoadBalancerLoadPlacementStrategy is illegal

6534756

eolivelli approved these changes Jul 6, 2022

View reviewed changes

Demogorgon314 reviewed Jul 7, 2022

View reviewed changes

conf/broker.conf Outdated Show resolved Hide resolved

better name

cf10e58

Demogorgon314 approved these changes Jul 7, 2022

View reviewed changes

Jason918 approved these changes Jul 8, 2022

View reviewed changes

codelipenghui added this to the 2.11.0 milestone Jul 11, 2022

codelipenghui added type/feature The PR added a new feature or issue requested a new feature area/broker labels Jul 11, 2022

heesung-sn approved these changes Jul 11, 2022

View reviewed changes

codelipenghui merged commit d5088e3 into apache:master Jul 12, 2022

dragonls mentioned this pull request Jul 12, 2022

[feature][broker] Add new load manager strategy for load manager to select broker having load below average. #14971

Closed

4 tasks

Jason918 mentioned this pull request Sep 4, 2022

[fix][broker]fix arithmetic exception for LeastResourceUsageWithWeight strategy #17149

Merged

2 tasks

dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022

[feature][broker]Provide new load balance placement strategy implemen…

e0c7710

…tation based on the least resource usage with weight (apache#16281)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature][broker]Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

[feature][broker]Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

HQebupt commented Jun 29, 2022 •

edited

HQebupt commented Jun 30, 2022

heesung-sn Jul 1, 2022 •

edited

HQebupt Jul 2, 2022

HQebupt Jul 2, 2022

heesung-sn Jul 2, 2022

HQebupt Jul 3, 2022

heesung-sn Jul 3, 2022 •

edited

dragonls Jul 4, 2022

heesung-sn Jul 7, 2022

eolivelli left a comment

HQebupt commented Jul 5, 2022 •

edited

HQebupt commented Jul 6, 2022

HQebupt commented Jul 6, 2022

eolivelli left a comment

HQebupt commented Jul 6, 2022

HQebupt commented Jul 7, 2022

HQebupt commented Jul 8, 2022

Jason918 left a comment

HQebupt commented Jul 8, 2022

HQebupt commented Jul 8, 2022

heesung-sn commented Jul 11, 2022

[feature][broker]Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

[feature][broker]Provide new load balance placement strategy implementation based on the least resource usage with weight #16281

Conversation

HQebupt commented Jun 29, 2022 • edited

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

HQebupt commented Jun 30, 2022

heesung-sn Jul 1, 2022 • edited

Choose a reason for hiding this comment

HQebupt Jul 2, 2022

Choose a reason for hiding this comment

HQebupt Jul 2, 2022

Choose a reason for hiding this comment

heesung-sn Jul 2, 2022

Choose a reason for hiding this comment

HQebupt Jul 3, 2022

Choose a reason for hiding this comment

heesung-sn Jul 3, 2022 • edited

Choose a reason for hiding this comment

dragonls Jul 4, 2022

Choose a reason for hiding this comment

heesung-sn Jul 7, 2022

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

HQebupt commented Jul 5, 2022 • edited

HQebupt commented Jul 6, 2022

HQebupt commented Jul 6, 2022

eolivelli left a comment

Choose a reason for hiding this comment

HQebupt commented Jul 6, 2022

HQebupt commented Jul 7, 2022

HQebupt commented Jul 8, 2022

Jason918 left a comment

Choose a reason for hiding this comment

HQebupt commented Jul 8, 2022

HQebupt commented Jul 8, 2022

heesung-sn commented Jul 11, 2022

HQebupt commented Jun 29, 2022 •

edited

heesung-sn Jul 1, 2022 •

edited

heesung-sn Jul 3, 2022 •

edited

HQebupt commented Jul 5, 2022 •

edited