Enable better ordering of ILM steps (allocate, migrate...) for more efficient resource usage

[tylerperk]

Description

Today, ILM steps run in a prescribed order which is designed to work ell for the typical/expected scenarios. For example, we expect replicas to be reduced or removed in later phases as less resiliency is needed for older data or once data is stored in Searchable Snapshots (Cold, Frozen). Thus, we perform allocate before migrate. Usually this is okay.

However, if a use case requires adding more replicas in a later phase, the fact that we operate in this order creates additional system and network load: Extra replicas are created on Hot (extra storage), and then immediately moved to Warm (extra network transfer). We should allow this use case to be more efficient by enabling the steps to occur in a more optimal order. Whether hat would be automatic or as a setting is to be determined.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)