Disk decider prevents allocation/fast recovery

[henningandersen]

If a node is above low disk threshold when being restarted (rolling restart, network disruption or crash), the disk threshold decider prevents reusing the shard content on the restarted node.

This seems unfortunate, in particular in the good case where we can do a noop recovery or an operations based recovery with only few operations (since that disk usage is already accounted for).

Notice that being above the low disk threshold on a node is not a bad state in itself. The cluster may have plenty space available, but an imbalance w.r.t. disk usage can make some nodes go above low threshold anyway. We only start moving shards off the node when high threshold is reached.

The test case here demonstrates this. It also demonstrates that even having nodes with space available for the shard is not enough due to delayed allocation.

This might partially repair itself (not demonstrated) in that when enough shards have been recovered elsewhere, the node could drop below the low threshold, making the rest of the shard contents available for faster recoveries.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Allocation)

[henningandersen]

We discussed this at our weekly sync and while we did not reach a conclusion, we did cover following:

• There were agreement that this is a problem today.
• The watermarks may exist primarily as a hysteresis when moving shards off nodes.
• We may want to consider adding a test framework for clusters in high disk usage situations.