Remove broker weights for ModularLoadManager #442
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… impact during rolling deployments
…okers are filtered out
…dles to brokers in ModularLoadManagerImpl
…entation Update ModularLoadManager.md
merlimat
reviewed
Jun 1, 2017
| brokerData.getLocalData().getWebServiceUrl(), totalMessageRateEstimate, weight, score); | ||
| return score; | ||
|
|
||
| log.debug("Broker {} has long term message rate {}", |
There was a problem hiding this comment.
typically we wrap all the debug statements within a if (log.isDebugEnabled()) {....} to avoid any parameters boxing or Object[] vargs allocations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When an overloaded broker is restarted, all the bundles hosted by that broker typically get assigned right back to the same broker by the newer
ModularLoadManagerImpl. The default (and only) strategy for this load manager isLeastLongTermMessageRate, which uses the combined long-term message rates (in+out), multiplied by the weight of the broker.The problem is the broker weight is determined by the inverse of how much headroom the broker has. More specifically, it's the inverse of
(overload_threshold - max_usage). In our production environment, we have noticed this weight varies from 3.2 to +Infinity (which happens when the broker is at or above the overload threshold). As a result, the load manager will assign bundles to the newly restarted broker until it has roughly triple the message rate of the least loaded broker. I.e., it will assign all the bundles that were unloaded right back to itself.Modifications
The weights were removed so the modular load manager places bundles based purely on the long-term message rates. It is expected that many (but not all) bundles will get assigned to a broker which was restarted. In the future, we would like to have static weights for each broker (with a default).
Result
No change if the older SimpleLoadManager is being used (which is the default). For deployments configured with the newer ModularLoadManager, load should be more evenly distributed based on long-term message rates.