ILM Freeze Phase - Freeze Option Outside of Cold Phase

[chirilloj]

Describe the feature:

Our Elastic environment is a hot-cold architecture (no warm). We keep indexes on hot nodes for 30 days and then transition them to cold nodes sith ILM. Indexes are then purged at 180 days. However, we also want to use ILM to freeze our indexes, but we are unable to with the current ILM design. We want to freeze our indexes at 90 days (through ILM), but the “freeze” option would freeze the indexes soon as they hit the cold nodes at 30 days – we don’t want this. We want a way to freeze our indexes at a different internal than when they are hit the cold nodes. We want to have indexes hit cold nodes in 30 days, and then have them frozen in 90 days, then deleted in 180 days.

Please add something like a “freeze phrase” that allows for a time specification to a freeze an index.

[dakrone]

Hi @chirilloj thanks for trying out ILM

In your case, can you describe why having a warm phase that does what the current cold phase does at 30 days, and then move the cold phase to have a min_age of 90d with the freeze action wouldn't work for you? Perhaps you can post your entire policy so we can get a better idea of what you are doing.

[elasticmachine]

Pinging @elastic/es-core-features

[chirilloj]

The ILM policy is below. We technically have a warm phase but all it does is a force merge of our data. We have our hot phase setup to roll to warm when an index reaches 50 gigs or is a max of 7 days old.

If we setup a warm phase at 30 days as suggested, our force merge of data is pushed out to 30 days (instead of 7 days or when we role by size), which we don't want - afraid of introducing problems into the environment if the force merge is pushed out that far. Thoughts? Let me know if you have further questions.

We could API force merge like we are with freeze now, but that is just swapping one for the other. The objective is to run all the pieces in ILM instead of having some in ILM and some through the API.

{
  "roll_weekly_delete_180": {
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            },
            "rollover": {
              "max_age": "7d",
              "max_size": "50gb"
            }
          }
        },
          "warm": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 50
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "set_priority": {
              "priority": 0
            },
            "allocate": {
              "require": {
                "box_type": "cold"
              },
              "exclude": {
              },
              "include": {
              }
            }
          }
        },
        "delete": {
          "min_age": "180d"
          "actions": {
            "delete": {
            }
          }
        }
      }
    }
  }
}

[Sakorah]

I would need the same functionality. I have hot and warm phase where the indices still are on the hot nodes and the cold phase to move them to the cheaper servers after a certain time. between the cold and delete phase I would like to define a freeze.

Right now I need to use a combination of ILM (handling rollover in hot, allocation in warm and delete) and curator (force_merge while index is still in hot).

A custom phase definition would be great!