ILM - Searchable snapshot action: gets stuck if there are two searchable snapshots with different types and ILM is being resumed

[gmarouli]

Elasticsearch Version

8.6.2+ (probably, 7.17.x+ too)

Installed Plugins

No response

Java Version

bundled

OS Version

N/A

Problem Description

Premise:

• An index is managed by an ILM policy which includes both a fully and a partially mounted searchable snapshot.
• The index is currently either fully or partially mounted.
• For some reason it loses the lifecycle execution metadata, for example the user used POST {index}/_ilm/remove.
• The user reapplies the same policy.

Expected:
As a user I expect ILM to pick up where it left off and continue with the execution of my policy. This is how it behaves if there is only one searchable snapshot action. The branch-check-prerequisites will see that the index is a searchable snapshot and it matches the repo, the type with the policy and it will skip the action.

Current behaviour:
ILM will pick up the mounted index move it through the steps, but when it hits the other searchable snapshot phase (aka the searchable snapshot will have a different type) it will fail with the following error:
index [restored-my-index-000001] in snapshot [found-snapshots/some_newer_snapshot] is a snapshot of a searchable snapshot index backed by index [my-index-000001] in snapshot [found-snapshots/the_snapshot_with_the_original_index] and cannot be mounted; did you mean to restore it instead?

The cause of this is that in the branch-check-prerequisites step checks if the index is a searchable snapshot using the index metadata but in branch-check-existing-snapshot we only check the lifecycle execution state which does not contain the information about the snapshot.

Steps to Reproduce

Add an ILM policy with searchable snapshot action in both cold and frozen:

PUT _ilm/policy/my-policy
{
	"policy": {
		"phases": {
			"frozen": {
				"min_age": "10m",
				"actions": {
					"searchable_snapshot": {
						"snapshot_repository": "found-snapshots"
					}
				}
			},
			"cold": {
				"min_age": "2m",
				"actions": {
					"searchable_snapshot": {
						"snapshot_repository": "found-snapshots"
					},
					"set_priority": {
						"priority": 0
					}
				}
			},
			"hot": {
				"min_age": "0ms",
				"actions": {
					"rollover": {
						"max_docs": 2,
						"min_docs": 1
					}
				}
			}
		}
	}
}

Add an index and insert some documents to trigger a rollover:

PUT _index_template/my-index
{
	"index_patterns": [
		"my-index-*"
	],
	"template": {
		"settings": {
			"index": {
				"lifecycle": {
					"name": "my-policy",
					"rollover_alias": "my-index"
				},
				"number_of_replicas": 0
			}
		}
	},
	"composed_of": []
}

PUT my-index-000001
{
	"aliases": {
		"my-index": {}
	}
}

POST _bulk
{ "index" : { "_index" : "my-index", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "my-index", "_id" : "2" } }
{ "field1" : "value2" }
{ "create" : { "_index" : "my-index", "_id" : "3" } }
{ "field1" : "value3" }

Verify with ILM explain that the index has been fully mounted and remove ILM:

POST my-index-000001/_ilm/remove

Then reapply the policy:

PUT restored-my-index-000001/_settings
{
	"index": {
		"lifecycle": {
			"name": "my-policy",
			"rollover_alias": "my-index"
		}
	}
}

ILM will pick up the index move it through the steps, but when it hits the frozen phase it will fail with the following error:
index [restored-my-index-000001] in snapshot [found-snapshots/some_newer_snapshot] is a snapshot of a searchable snapshot index backed by index [my-index-000001] in snapshot [found-snapshots/the_snapshot_with_the_original_index] and cannot be mounted; did you mean to restore it instead?

The cause of this is that in the branch-check-prerequisites step in the cold phase we check if the index is a searchable snapshot using the index metadata but in branch-check-existing-snapshot in the frozen phase we only check the lifecycle execution state which was not updated before because the action was skipped.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[abertolli]

Thanks for writing this report. We have also been seeing some strange things with regards to indices. We have a relatively new cluster on 8.6.2 and just recently added a frozen tier. When we did, we immediately saw partial-restored- (partially mounted/fully mounted hybrids?) indices which appear to have been moved from cold to frozen.

Does this sound like part of the same problem? Is it ok for an index ever be named partial-restored-indexname?

[gmarouli]

@abertolli I have seen partial-restored-* before but we would need to see what happened in your case to confidently say what's going on. Can you reproduce it with a minimal example?

[abertolli]

@abertolli I have seen partial-restored-* before but we would need to see what happened in your case to confidently say what's going on. Can you reproduce it with a minimal example?

I re-read the bug report, and it seems you already know what is causing the problem you are seeing above. Although we also have that issue (and that error) on a cluster, I think the issue I'm talking about might be separate. If it is related, it's not clear to me yet. But I will definitely look forward to seeing this bug corrected.

[gmarouli]

@abertolli I did check on a recent version and when an index transitions from cold with a fully mounted searchable snapshot to frozen with a partially mounted it ends up with that prefix. So, I do not think it's signalling that something went wrong

[gmarouli]

We include this scenario in #102705, which is a more complete approach to make ILM resilient when it is freshly applied to a searchable snapshot.