support ilm phase timings based on index naming date_math rather than creation_date

[bczifra]

Describe the feature:
Sometimes indices are named based on the dates that their documents pertain to, not based on the date the indices are created. As an example, on May 23 an index could be created with a name like foo-2019.05.20.

It would be be helpful if the ILM phase timings could be based on the index naming date math rather than the index's creation_date, supporting options like:
"for this monthly index, keep 3 months of data"
"for this weekly index, keep 12 weeks of data"
"for this daily index, keep 20 days of data"

In each of these situations, the start date of that elapsed time period should be calculated based upon the date specified in the index name.

The Rollover API does support index name date math, but ILM does not.

[elasticmachine]

Pinging @elastic/es-core-features

[dakrone]

The Rollover API does support index name date math, but ILM does not.

This isn't right, ILM does support index name date math just like rollover does, it requires that the index name be created with date math, for instance:

PUT _ilm/policy/roll
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "10s"
          }
        }
      }
    }
  }
}

PUT _template/roll
{
  "index_patterns": ["roll-*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index.lifecycle.name": "roll",
    "index.lifecycle.rollover_alias": "roll-alias"
  }
}

PUT /_cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "10s"
  }
}

PUT /%3Croll-%7Bnow%2Fs%7Byyyy-MM-dd-HH-mm-ss%7D%7D-1%3E
{
  "aliases": {
    "roll-alias":{
      "is_write_index": true
    }
  }
}

Creates indices that follow the date math as they are rolled over.

[dakrone]

We have a plan for addressing situation like this, I'll outline the general plan (which may shift as implementation proceeds).

Step one is to add a setting, index.lifecycle.origination_date that a user can set on an index, if this is set, ILM will use this date to calculate the index age for its phase transitions. This allows a user to manually create an "old" index with an old origination date when indexing older data. This custom origination date should also be exposed in the ILM explain output for an index.

Step two is to add another index setting allowing the origination date setting to be automatically configured on index creation based on some pattern in the index name. We haven't yet decided how to parse the index name, or whether this should be template level or index level. This would allow us to automate setting the origination date for newly created indices containing older data.

@andreidan is going to start working on step one.