Auto-expand-replicas causes problems in a small cluster rolling restart

[DaveCTurner]

Many indices (particularly system indices) use auto_expand_replicas: 0-1 to avoid unassigned shards in a one-node cluster. However, when restarting a node in a cluster with just two nodes in a data tier this setting causes undesirable behaviour:

• the cluster may remain in green health even though really it has unassigned shards and is running with less resilience than intended

• replicas on the restarting node are effectively destroyed (at least, their retention leases are dropped) which forces a full file-based recovery

There's room for improvement here. Could we use the node-shutdown or desired-nodes features to report accurate health, and avoid file-based recovery, for auto-expand replicas indices while a node is restarting?

---

Workaround

If you have a small cluster but never intend to shrink its hot/warm/content tiers to a single node, set number_of_replicas: 1 instead of using auto_expand_replicas.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)