Prevent deletion of snapshots that are mounted and ILM delete action

[tlrx]

Today nothing prevents the deletion of a snapshot that is used by a searchable snapshot index. If such a snapshot is deleted the mounted index will go RED in the future. We started to implement a check in #73821 to prevent the deletion of snapshots that are used by searchable snapshots indices within the same cluster and it made some ILM tests to fail.

ILM has a delete index action DeleteAction with a delete_searchable_snapshot flag that, when set to true (the default value), deletes the snapshot that is mounted before deleting the index. ILM deletes the snapshot before the index because the information about the snapshot are stored in the index's metadata (under the ILM custom metedata). If ILM was deleting the searchable snapshot index before the snapshot it would not know which snapshot to delete.

I still think it is important to prevent the deletion of mounted snapshots so we should investigate how to keep the ILM behavior while implementing a safety check in #73821.

I can see different solution and I'm opening this issue to discuss them:

• Adds a specific flag to the delete snapshot request that allows to bypass the safety check. This flag should not be exposed at the REST API level and can be kept as internal, only set by ILM delete action.
• Allow to delete a searchable snapshot index and its snapshot together. The index metadata should be removed from the cluster state at the same time the snapshot is added as a snapshot deletion in progress. It implies multiple verifications (if the snapshot is used by another index, if the snapshot is used by a restore etc) but it should be doable. Wether this should be implemented in the existing delete index action, the existing snapshot delete action or a new action (like Unmount API) is left to decide.
• ILM keeps the information of snapshots to delete somewhere else

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[tlrx]

@henningandersen suggested the idea to add a flag to the existing Mount API to indicate that the index "owns" the snapshot and that it should be deleted when the searchable snapshot index is deleted. I'm going to give it try.