[ILM] The downsample action runs before the allocate and migrate actions

[andreidan]

Elasticsearch Version

8.7+

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin

Problem Description

In the warm and cold phases the downsample action runs before allocate and migrate https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/TimeseriesLifecycleType.java#L69

This means that if I define a policy with hot and warm like so:

      "phases" : {
        "hot" : {
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d",
              "max_docs" : 2000000000
            }
          },
        "warm" : {
          "min_age" : "2d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 0,
              "include" : { },
              "exclude" : { },
              "require" : { }
            },
            "downsample" : {
              "fixed_interval" : "10h"
            }
          }
        }

And index would start in hot, wait to meet the rollover condition, rollover, and then while still in the hot tier it will
run the downsample action. Note that the source index will still be in the hot tier whilst downsample runs. (the downsample index will also be in hot for reasons detailed here: #96733)

After downsample is completed, the downsampled index will migrate to the warm tier.

Note that this essentially makes the downsample action definition in the ILM hot and warm phase identical from an execution perspective (there's no difference as to which tier downsample will run on).

Similarly in a deployment with hot/warm/cold tiers, if downsample is defined in the cold phase the operation will run on the warm tier (i.e. source index is in warm, runs downsample, when downsample is completed it migrates to cold - as it's defined to run before the migrate action in cold https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ilm/TimeseriesLifecycleType.java#L79 )

This might be desired in which case we should document this behaviour as it's a bit surprising. But maybe downsample should run after the migrate action?

Steps to Reproduce

Run downsample with ILM and check the logs for the source index i.e notice the downsample-* index is created whilst the source index is in the tier prior to the tier where the downsample action is defined
https://www.elastic.co/guide/en/elasticsearch/reference/current/downsampling-ilm.html

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[andreidan]

I've tagged ILM and TSDB on this issue (as ILM is where possible changes would need to be made) however it'd be up to the @elastic/es-analytics-geo team to define the desired behaviour here.

[leighatelastic]

Can we have an update on this item please?

[martijnvg]

@andreidan I think it makes sense that the downsample action runs after allocate and migrate actions. But I recall this requires a breaking change in ILM? Additionally is this more a storage engine task or a data management task?

[andreidan]

@martijnvg yeah, we have a PR out there that we need to pick up and drive to completion #97745

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)