New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disk allocation not periodically checking for whether thresholds/watermarks are exceeded #8146
Comments
I agree this is not how it should be... I think it would make sense to issue reroute commands from the |
@s1monw I think it should be implemented differently. I think the I think a better way to implement it would be to allow the What do you think about this? |
yeah I think that is the right way to do it abstraction wise... |
This adds a Listener interface to the ClusterInfoService, this is used by the DiskThresholdDecider, which adds a listener to check for nodes passing the high watermark. If a node is past the high watermark an empty reroute is issued so shards can be reallocated if desired. A reroute will only be issued once every `cluster.routing.allocation.disk.reroute_interval`, which is "60s" by default. Refactors InternalClusterInfoService to delegate the nodes stats and indices stats gathering into separate methods so they have be overriden by extending classes. Each stat gathering method returns a CountDownLatch that can be used to wait until processing for that part is successful before calling the listeners. Fixes elastic#8146
This adds a Listener interface to the ClusterInfoService, this is used by the DiskThresholdDecider, which adds a listener to check for nodes passing the high watermark. If a node is past the high watermark an empty reroute is issued so shards can be reallocated if desired. A reroute will only be issued once every `cluster.routing.allocation.disk.reroute_interval`, which is "60s" by default. Refactors InternalClusterInfoService to delegate the nodes stats and indices stats gathering into separate methods so they have be overriden by extending classes. Each stat gathering method returns a CountDownLatch that can be used to wait until processing for that part is successful before calling the listeners. Fixes #8146
This adds a Listener interface to the ClusterInfoService, this is used by the DiskThresholdDecider, which adds a listener to check for nodes passing the high watermark. If a node is past the high watermark an empty reroute is issued so shards can be reallocated if desired. A reroute will only be issued once every `cluster.routing.allocation.disk.reroute_interval`, which is "60s" by default. Refactors InternalClusterInfoService to delegate the nodes stats and indices stats gathering into separate methods so they have be overriden by extending classes. Each stat gathering method returns a CountDownLatch that can be used to wait until processing for that part is successful before calling the listeners. Fixes #8146
This adds a Listener interface to the ClusterInfoService, this is used by the DiskThresholdDecider, which adds a listener to check for nodes passing the high watermark. If a node is past the high watermark an empty reroute is issued so shards can be reallocated if desired. A reroute will only be issued once every `cluster.routing.allocation.disk.reroute_interval`, which is "60s" by default. Refactors InternalClusterInfoService to delegate the nodes stats and indices stats gathering into separate methods so they have be overriden by extending classes. Each stat gathering method returns a CountDownLatch that can be used to wait until processing for that part is successful before calling the listeners. Fixes elastic#8146
Repo video below.
In short:
You will see that there is an index with shards allocated to 2 nodes. Cluster setting has watermark.low set to 80% and high set to 90%. Disk usage for node_local is at 76%, node_vm is at 83%.
You can skip the video from 0:30 to 2:38, that is when I incrementally add large files to the disk on node_vm so that it uses > 90% of disk. After 2:38, You will see that the disk usage has increased to 90%+ but no relocation of shards occurred from node_vm to node_local. But if I wait 30+s after the disk hits 90%+ usage, and then set the same cluster.routing.allocation.disk.* settings again, the relocation immediately happens. This suggests that we are not currently periodically checking for whether the disk is filling up beyond the high mark without manual intervention by either:
POST /_cluster/reroute
https://drive.google.com/file/d/0BzqaicoBfqMfdXdSWnI2OFp0eXM/view?usp=sharing
Workaround is to periodically call reroute command (or reset the same settings).
The text was updated successfully, but these errors were encountered: