[plugin/apm-data] Lifecycle for APM-Server datastreams switches to Unmanaged on upgrade

[lahsivjar]

Elasticsearch Version

main (> v8.15.1)

Installed Plugins

No response

Java Version

bundled

OS Version

All

Problem Description

APM-Server (serverless is not impacted) switched from Index Lifecycle Management(ILM) to Datastream Lifecycle (DSL) in v8.15.0. This switch was done when we moved from APM integration (which used ILM) to APM-data plugin (which uses DSL) in ES for managing APM datastreams. As a result of the switch, any old datastreams created before the switch would be Unmanaged because the datastream will never be updated with the DSL lifecycle -- this has to be done manually by using the PUT API.

To fix this, v8.15.1 has a patch to fallback to ILM if DSL is not defined. However, this patch needed to be reverted in main due to ILM not being supported for serverless offering.

Steps to Reproduce

1. Create a cluster with APM integration installed for versions before v8.15.0
2. Send some data to APM
3. Update the cluster to main
4. Send some more data to APM
5. Perform a rollover
6. Observe that the latest index in the datastream is Unmanaged

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)