Searchable snapshot should not be able to be deleted if it's in use by a frozen index

[kunisen]

Elasticsearch Version

8.13.3

Installed Plugins

No response

Java Version

bundled

OS Version

major

Problem Description

We found a doc bug for searchable snapshot. (cc @geekpete)

Description

The searchable snapshot doc says this:

You also cannot delete a snapshot if any of its indices are mounted as a searchable snapshot in the same cluster.

But we found actually we can delete even it’s mounted. Please see the screenshot.

Our concern

:: [1]
It literally means we have no protection to searchable snapshot, even it’s being mounted to frozen indices. People have deletion permission can delete it.
(We are not talking about deleting the frozen index, but only talking about deleting searchable snapshot directly)
Can we add this guardrail logic to make this statement become really true, or at least remove this part?

:: [2]
Also, if a customized SLM wants to delete searchable snapshot, it should error out too, based on the same context.

Our testing result:

Reference

• Clarify searchable snapshot repository reliability #93023: Seems like this info piece is added from 8.6
• Prevent deletion of snapshots that are used by snapshot backed indices #73821: But the corresponding PR is not merged
• Add snapshots pending deletion in cluster state to delete snapshot once index is deleted #79156: Another one superseded it but is not merged too

Steps to Reproduce

Prepare frozen index and searchable snapshot

PUT _cluster/settings
{"persistent":{"indices.lifecycle.poll_interval":"10s"}}

# create ilm policy for frozen
DELETE _ilm/policy/test-frozen-policy
PUT _ilm/policy/test-frozen-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0m",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "frozen": {
        "min_age": "1m",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      }
    }
  }
}


# create template for frozen indices
DELETE _template/test-frozen-template
PUT _template/test-frozen-template
{
  "index_patterns": ["test-frozen-index*"],
  "settings": {
    "lifecycle": {
      "name": "test-frozen-policy"
    },
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}

# create index for frozen
DELETE test-frozen-index-1
POST test-frozen-index-1/_doc
{
  "field1": "someValue"
}

Check searchable snapshot is created and try delete it

# this is the frozen index
GET partial-test-frozen-index-1/_settings

GET _snapshot/found-snapshots/2024.05.09-test-frozen-index-1-test-frozen-policy-tsqt9tzqtyququi2qcdohw

# response - searchable snapshot is in place and mounted to frozen index
{
  "snapshots": [
    {
      "snapshot": "2024.05.09-test-frozen-index-1-test-frozen-policy-tsqt9tzqtyququi2qcdohw",
      "uuid": "Jxzs_GCNTy-e8uHs00MqQQ",
      "repository": "found-snapshots",
      "version_id": 8503000,
      "version": "8503000",
      "indices": [
        "test-frozen-index-1"
      ],
      "data_streams": [],
      "include_global_state": false,
      "state": "SUCCESS",
      "start_time": "2024-05-09T07:24:22.541Z",
      "start_time_in_millis": 1715239462541,
      "end_time": "2024-05-09T07:24:22.941Z",
      "end_time_in_millis": 1715239462941,
      "duration_in_millis": 400,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      },
      "feature_states": []
    }
  ],
  "total": 1,
  "remaining": 0
}


DELETE _snapshot/found-snapshots/2024.05.09-test-frozen-index-1-test-frozen-policy-tsqt9tzqtyququi2qcdohw

# response - but we can delete it
{
  "acknowledged": true
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Pinging @elastic/es-docs (Team:Docs)