elastic - allow specifying index life cycle management policy

[assaf79]

The current implementation creates time based indices.
It would help if a there was a parameter that creates an ILM policy, so indices would be rolled over and deleted automatically.

[Lumotheninja]

+1 on this one. According to Elastic docs, it requires a number at the end which can be incremented by 1 at each rollover: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

I'm using Micrometer 1.2.0 and it doesn't have the option to append the -0000N at the end

[pmioulet]

In addition if indexes are not purged at some point, ES will hit it's internal shard limit and metrics can no longer be created :(
this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open

[shakuzen]

Metrics data retention policy is left to out-of-band configuration in the backend. If there is something that Micrometer can do to better enable index lifecycle management when using the micrometer-registry-elastic module, we can look at making that change. I agree having an index life cycle management policy for your metrics time series data sounds like good practice, but I'm not sure about configuring that client-side. Usually retention policies are more centrally controlled on the backend. I would need to look into this more to understand what changes might make sense, if any.

Can a policy not be made to delete the currently made daily indices older than some amount? Is the missing piece the rollover if a daily index gets too big?

[pmioulet]

Every time you create a new index you'll be creating new shards. At some point (10 months of daily rollover in our case because we have other indexes rolling over) you will hit the max shard index limit which is per ES node. You actually need to delete old indexes at some point. If the cleanup has to be handled externally (by curatore for ex.), it's fine but i reckon it could be highlighted in the docs maybe.

If the lib handles the rollover(which is an action that could also be done by the backend) it could also allow to setup a max index count or max history limit and handle the deletion in the lib.

[shakuzen]

If the cleanup has to be handled externally (by curatore for ex.), it's fine but i reckon it could be highlighted in the docs maybe.

I agree it could be worth a mention in our docs. This is the source for the elastic meter registry docs. If you have suggestion for wording to add, feel free to send a pull request, or open an issue in the repository so we can track updating the docs.

If the lib handles the rollover(which is an action that could also be done by the backend) it could also allow to setup a max index count or max history limit and handle the deletion in the lib.

This can seem reasonable, but it is probably a shared metrics index across many applications. We wouldn't be able to pick one policy that works for every user. And if it is configurable, each application could be configured differently. Then the result becomes unexpected for someone using the index. From an operations/management perspective, it seems better for this to be managed on the elasticsearch backend side.