Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed improvements for BalancedShardsAllocator #15678

Merged
merged 11 commits into from Dec 28, 2015

Conversation

Projects
None yet
4 participants
@ywelsch
Copy link
Contributor

commented Dec 28, 2015

This PR contains some speed improvements for BalancedShardsAllocator (relates to #6372):

  • Removal of the pattern node.addShard() -> calculate weight -> node.removeShard() which is expensive as, beside map lookups, it invalidates caching of precomputed values in ModelNode and ModelIndex. Replaced by adding an additional parameter to the weight function which accounts for the added / removed shard.
  • For rebalancing an index, only consider nodes that currently have a shard of that index or where the index can be allocated. This allows to prune a large number of nodes in case of hot/warm setup. Implemented by adding allocation decider rule based on IndexMetaData.
  • Removed some dead code
  • Some caching (avoid reresolving values in map all the time) / simplification of code

I made smaller commits for each improvement to simplify the review process.

@ywelsch ywelsch added review labels Dec 28, 2015

@s1monw

View changes

...ain/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java Outdated
final float weightIndex = (node.numShards(index) - balancer.avgShardsPerNode(index));
assert theta != null;
return theta[0] * weightShard + theta[1] * weightIndex;
public float weight(Balancer balancer, ModelNode node, String index, int numAdditionalShards) {

This comment has been minimized.

Copy link
@s1monw

s1monw Dec 28, 2015

Contributor

can we instead of exposing this additional parameter just have some dedicated methods for +1, -1, 0 and keep that method internal?

@s1monw

This comment has been minimized.

Copy link
Contributor

commented Dec 28, 2015

left one comment - LGTM otherwise

ywelsch added some commits Dec 28, 2015

Eliminate adding/removing shard to simulate weight of added shard / r…
…emoved shard

Removal of the pattern node.addShard() -> calculate weight -> node.removeShard() which is expensive as, beside map lookups, it invalidates caching of precomputed values in ModelNode and ModelIndex. Replaced by adding an additional parameter to the weight function which accounts for the added / removed shard.
For rebalancing an index, only consider nodes that currently have a s…
…hard of that index or where the index can be allocated

This allows to prune a large number of nodes in case of hot/warm setup

@ywelsch ywelsch force-pushed the ywelsch:enhance/faster-balancer branch to 7e134da Dec 28, 2015

@ywelsch

This comment has been minimized.

Copy link
Contributor Author

commented Dec 28, 2015

Addressed comment and added labels. Thanks for the review!

ywelsch added a commit that referenced this pull request Dec 28, 2015

Merge pull request #15678 from ywelsch/enhance/faster-balancer
Speed improvements for BalancedShardsAllocator

@ywelsch ywelsch merged commit 24ab66c into elastic:master Dec 28, 2015

1 check passed

CLA Commit author is a member of Elasticsearch
Details
@clintongormley

This comment has been minimized.

Copy link
Member

commented Jan 10, 2016

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.