Depending on how you have ES nodes allocated across the system, and the way the your data goes across indices, it is possible to end up with a cluster that is weighted (in terms of disk) towards certain physical nodes. ES-equilibrium was created to ensure that the disk limit wasn't accidentally reached when creating new shards, and to allow the ability to balance disk usage across nodes in the cluster.
A good rule of thumb is: If you don't know whether you need this, you probably don't need it.
The elasticsearch-equilibrium plugin can be installed as any other ES plugin using bin/plugin utility:
% bin/plugin -url https://oss-es-plugins.s3.amazonaws.com/elasticsearch-equilibrium/elasticsearch-equilibrium-0.90.0.zip -install elasticsearch-equilibrium-0.90.0
sonian.elasticsearch.equilibrium: enabled: true minimumAvailablePercentage: 20.0 minimumSwapDifferencePercentage: 5.0 minimumSwapShardRelativeDifferencePercentage: 75.0
This plugin currently keeps ES from allocating shards to a node with less than minimumAvailablePercentage percentage available disk space on the ES data mount points.
If a node goes beneath this amount of free disk space percentage, no more shards will be relocated or assigned to this node. Defaults to 20.0%
If two nodes have a percentage difference of at least this amount in used disk space, a shard swap will be initiated. Defaults to 5.0%
Minimum difference in percentage between two shards to consider them viable for swapping. For example, if set to 50.0%, the smaller shard must be <= 50% the size of the largest shard candidate for swapping. Defaults to 75.0%
% curl -s "http://localhost:9200/_rebalance"
Manual rebalancing looks through the cluster to discover the node with the highest disk usage (as a percentage), and the node with the lowers disk usage (also as a percentage). It then finds the largest shard on the largest node and swaps it with the smallest shard on the smallest node.
In this way, disk usage that may be uneven across a cluster can be evened out.
Note: this should only be done under supervision of someone that knows what they are doing, and only one rebalance should be run at a time. Wait for the rebalance to finish before starting another.
Note that elasticsearch-equilibrium will fail open in the event a nodeFsStats request times out.
If you need to add elasticsearch-equilibrium to your pom.xml (for some reason), you can use:
<dependency> <groupId>com.sonian</groupId> <artifactId>elasticsearch-equilibrium</artifactId> <version>0.90.0</version> </dependency>
Copyright 2012 Lee Hinman & Sonian
Released under the Apache License, 2.0. See LICENSE.txt