Index Settings: Auto expand replica enabled indices ignore number of replica updates without telling the user

[spinscale]

Elasticsearch version (bin/elasticsearch --version): 6.0.0

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Ran this on a two node cluster (so auto expanding works)

DELETE foo

PUT foo
{
  "settings": {
    "index.auto_expand_replicas" : "0-all"
  }
}

GET _cat/shards

# this is acknowledged, but the setting is not updated due to the auto expanding replicas
PUT foo/_settings
{
  "settings": {
    "index.number_of_replicas" : 2
  }
}

# returns 1
GET foo/_settings?filter_path=**.number_of_replicas

# state is green, as auto expand setting is still in place
GET _cat/indices/foo

The behaviour looks correct to me, but returning an error on the update indices settings call makes sense to me.

[pickypg]

I kind of wonder if we should just deprecate auto_expand_replicas and add its functionality to number_of_replicas instead?

[GlenRSmith]

@pickypg If we did that, what would be reported on inspection of the index settings number_of_replicas?

[pickypg]

@GlenRSmith The value that is set, which could continue to default to 1, or possibly we could change the default to "0-1". Because it's just a setting and not a live representation of how many replicas are allocated, I don't think there's much risk with adding functionality to it by merging it with auto_expand_replicas.

[GlenRSmith]

@pickypg Currently, if you set auto_expand_replicas, number_of_replicas will report the current target count. If these are merged, number_of_replicas will have to report either the expand setting or the target, as is currently reported, so it seems to me that such a change will result in information loss.

[pickypg]

@GlenRSmith I think it's a bug that the value reported by number_of_replicas changes. Under normal usage of number_of_replicas, it doesn't change based on how the cluster is actually behaving and I cannot think of any other setting that behaves that way either. Checking the _settings for state seems like a very bad idea to me.

It's easy to see the number of allocated replicas via /_cat/indices or /_cat/shards (or even just /_shard_stores for unallocated shards and specifying the ?status=green as desired)

[PnPie]

I see that when auto_expand_replicas is enabled, the way we achieve "automatically update replicas number" is to re-set it's number_of_replicas settings (IndexMetaData.SETTING_NUMBER_OF_REPLICAS) and then kick off a reroute. So this is why we get the "current replicas number state" while hitting _settings, because the setting itself changes all the time when auto_expand_replicas enabled.

So I agree with @pickypg and I also think it's better to merge auto_expand_replicas into number_of_replicas as these 2 settings are kind of one thing and especially when auto_expand_replicas enabled, seems number_of_replicas doesn't make any sense.

[gwbrown]

I'm labeling this team-discuss because the problem is clear, but it does not seem that a solid conclusion was reached about what to do about it.

[rjernst]

I'm relabeling this issue for the distributed team to decide any future action on, since this is not a settings infrastructure issue.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)