Thanks for opening this issue! I think we will get to this pretty soon ie. next week or so

eta?

@dakrone what's the status of this...

@synhershko I'm currently working on developing this, it's trickier than a usual AllocationDecider (which prevents allocation) because fetching the disk usages and shard sizes is overhead that we don't want to incur for every operation, so it needs to be cached for a time and refreshed at certain intervals.

hey guys - any chance this piece of decider will participate in other operations working with the FS, like optimization for example? what we are seeing is large indices being optimized and occasionally servers running very low on disk space because of that.

Maybe if an index doesn't have enough room to optimize a rebalancing should kick in?

Hmm that is something that is pretty rare condition though. I wonder if we really should have something like this in the core system or if we should just ask for a customer allocation decider since deciders can trigger a rebalance on such a condition via canRemain - maybe this should go in a different issue? Can you open one?

#3807

It isn't this rare if you run a large data shop with replicas and all, with data constantly going in. It doesn't happen everyday, but it did happen to us.

IMO optimize should be rare in most cases unless you have time based indices etc. ;)

We do use rolling indexes...

Are you seeing a consistent amount of disk used for the optimize? If you know in advance about how much room you'll need for the optimize, you could set the high watermark for the disk threshold and ES should relocate shards if the disk usage passes that watermark.

Since ES can have this kind of info and do the maths for me, I don't see why I need to plan for it in advance. Plus I don't think ES can relocate a shard which is in the middle of it being optimized, and setting too high high-watermark is something we wouldn't want to do as well.

@dakrone something that just occurred to us - how would the free space decider play along with ES's defaut to try and have the same number of shards on each node?

In our scenario (and I'm assuming this is quite common) we have many data servers each with different HD capacities, ranging from ~120GB to ~1000GB. I'm pretty sure if ES will try to balance based on both criteria something will go very wrong.

Did you take that into account? or should we try breaking this with some nasty tests?

@synhershko since the decider is part of the balancing process, the allocator will attempt to find the "best" weights that still satisfy all of the deciders, so it will try to balance the shards evenly, but will still allow uneven allocation in the event that the disk limit has been reached on a particular machine or set of machines.

Did you take that into account? or should we try breaking this with some nasty tests?

It should already be taken into account, but nasty tests are always appreciated! :)

Will try to get to it soon, then

@dakrone just to let you know we now use this feature in our highly uneven cluster and so far all looks good. It seems like ES still tries to even the number of shards on each node but the free-space decider seems to do a good job. Thanks!

Awesome! I'm glad to hear it's working well for you! :D

@synhershko this is great that you come back to us. Do you have trouble with the balancing in anyway as you mention that it tries to balance?

@s1monw no it all seems to be fine. What I meant is when there's balancing in action, which hardly ever happens now as far as I can tell, it will try to get to an end result where disk allocation limits are respected AND there is more or less the same amount of shards on each node. Which I think makes sense.

Before upgrading I set out to write some integration which will test our cluster configuration with the internal moving pieces. I was able to recreate a similar scenario in the test (different node sizes, different index sizes) and everything worked (shards were allocated, no node was over-allocated etc). I just couldn't find edge-cases to test there - all seems to have already been tested in @dakrone 's tests so we dropped this effort and decided to take the pill.

We upgraded from a variant of 0.90.0 (custom compiled with some mods) to vanilla 0.90.7. The upgrade took a while (different Lucene versions) but went smooth, and once the disk-aware decider was enabled it took the cluster some while to stabilize, but ever since we did that all seems to work fine.

I'm leaving the company this week so will probably stop monitor that cluster, but as I said so far this looks very good and stable.