[Index Template] Use index.auto_expand_replicas '0-1'

[pickypg]

The Beats index / indices should consider using the index.auto_expand_replicas setting, set to "0-1" in favor of index.number_of_replicas set to 1 (or leaving it as the default).

Doing so will allow single node (e.g., dev) clusters to startup as green, which gives a better getting started experience.

[ruflin]

@pickypg Interesting, wasn't aware of this option. At the moment we don't set it at all (I think) and use the default from ES. Do we have to change anything in this case?

[pickypg]

@ruflin The default value for the setting is false, which disables it. The change on the Beats template would be to add auto_expand_replicas as an extra setting to the template. For example:

"settings": {
  "index.auto_expand_replicas": "0-1"
}

As with all index.* settings, you can choose to prefix it with index. or simply drop it (e.g., just auto_expand_replicas). FYI, when auto_expand_replicas is set, number_of_replicas is ignored (elastic/elasticsearch#27835). I'm hoping that we simply merge the two settings in ES.

[ruflin]

Great. It's pretty simple for us to add such settings to our config like we do here for shards: https://github.com/elastic/beats/blob/master/filebeat/filebeat.yml#L81 It would be nice if the two could be merged because I worry that people will try to set both.

[ruflin]

@elastic/beats After a conversation with @pickypg I'm thinking we should add this to our short config for a better getting started experience, similar as we do with shards. This means it's not hardcoded anywhere in our binary but only exists in our filebeat.yml file for example.

This means if someone just starts a beat against a 1 elasticsearch node to tests, he will have a green cluster state. As soon as someone scales up he should change this config to his own needs.

Having it in our short config I would also not treat it as a breaking change as everyone that keeps the old config still has the same behaviour. If a beat of the exact same version with the config and without the config exists, the first one "wins", means writes the template. We have the same problem with number of shards and so far this didn't seem to be an issue.

@graphaelli I would recommend to do the same for apm-server.

[graphaelli]

@ruflin Thanks for the reminder, getting this in now.

[urso]

@ruflin @pickypg What happens if the user still has the setting set from testing and now deploys with a bigger cluster of N nodes (without changing the auto_expand_replicas setting)? Can this potentially mark a node green, while it should be yellow?

TBH I'm not convinced by: User just has to change this setting in beats when going with a bigger cluster. The main problem is not the potential race on template loading, but the user not being aware of this setting (who reads documentation?).

[ruflin]

@urso For yellow instead of green, yes. For the second part, not reading docs, agree. I would hope someone that goes into production would revise the config settings like shards, replicas and auto-expand but definitively will be skipped form time to time. We should probably add a note in the config file, that this should not be enabled for production clusters.

[ruflin]

@pickypg @graphaelli We discussed this again in the beats team sync and decided to not move forward with this as we worry about the cases where people take this into production and have a green cluster when they shouldn't.