DIsk based allocator should take into account the amount of disk space which can be reused

[maf23]

Assume we have a node where the disk usage is above the low watermark. If we restart this node then no shards will be allocated on it since the disk usage is above the low watermark.

It would be better if the disk based allocator could realize that it can throw away or at least reuse the space which is currently used for outdated shard copies on the node.

[clintongormley]

@dakrone any thoughts about this?

[dakrone]

@clintongormley hmm.. to implement this we would have to send a request for each shard to all nodes to see what files and checksums they have in common. This would be quite heavy, especially for an AllocationDecider (because it has to run in tight loops).

[clintongormley]

@dakrone thanks for commenting, that sounds... heavy. Perhaps once sequence IDs (#10708) are in, it'll be more feasible.

Marking as stalled.

[clintongormley]

@dakrone @bleskes I think this has been resolved already, no?

[dakrone]

@clintongormley no, it hasn't be resolved still, I'm not sure if the seqids work is far enough along to work on this, maybe @bleskes knows?

[bleskes]

I don't think we can expect this in the first iteration. The problem is very similar to the current file based copy - we don't know how much will be reused (or if seq no base recovery will succeed) until we start the work. The master needs to know this in advance somehow (as Lee already said).

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

As I understand it, the situation described here is a transient problem around a node restart in the sense that the cluster sorts itself out eventually once it works out what data it no longer needs. As such, since its opening we have not seen enough feedback that it is something we should work on. We prefer to close this issue as a clear indication that we are not going to work on this at this time. We are always open to reconsidering this in the future based on compelling feedback; despite this issue being closed please feel free to leave feedback (including +1s).