Unable to mount searchable snapshot for backing indices of `.fleet-actions-results` system data stream

[romain-chanu]

Elasticsearch Version

8.15.0

Installed Plugins

No response

Java Version

bundled

OS Version

ESS

Problem Description

Issue observed in the field

Users can edit the managed ILM policy .fleet-actions-results-ilm-policy to include a frozen phase.

When a backing index from the .fleet-actions-results enters the frozen phase, Elasticsearch is unable to mount the respective searchable snapshot.

Such error message is observed in the ILM explain result:

        "ilm": {
          "phase": "frozen",
          "failed_step": "mount-snapshot",
          "phase_definition": "{\"policy\":\".fleet-actions-results-ilm-policy\",\"phase_definition\":{\"min_age\":\"0d\",\"actions\":{\"searchable_snapshot\":{\"snapshot_repository\":\"found-snapshots\",\"force_merge_index\":true}}},\"version\":2,\"modified_date_in_millis\":1733296846017}",
          "snapshot_repository": "found-snapshots",
          "snapshot_name": "2024.12.04-.ds-.fleet-actions-results-2024.12.04-000001-.fleet-actions-results-ilm-policy-6i5yigyat6cymxkb7ub37g",
          "phase_time": "1733297097263",
          "step_info": "{\"type\":\"illegal_argument_exception\",\"reason\":\"requested system indices [.ds-.fleet-actions-results-2024.12.04-000001], but system indices can only be restored as part of a feature state\"}",
          "creation_date": "1733297037388",
          "step_time": "1733297097463",
          "action_time": "1733297038142",
          "is_auto_retryable_error": "true",
          "failed_step_retry_count": "3",
          "snapshot_index_name": ".ds-.fleet-actions-results-2024.12.04-000001",
          "action": "searchable_snapshot",
          "step": "ERROR"
        },

This opens the conversation around searchable snapshots and system data streams/indices (part of feature states) and whether we should even allow users to configure cold/frozen phases for ILM policies that are potentially attached to these.

Could the team review this? Do we consider this as an issue and do we want to allow / restrict such use case?

Thank you team 🙏

Steps to Reproduce

1. Create an ESS cluster with 1 hot node, 1 frozen node, one Integrations Server

2. Create a role to access .fleet* indices:

POST _security/role/fleet_superuser
{
  "indices": [
    {
      "names": [
        ".fleet*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

3. Create a user with that role:

POST _security/user/temp_user
{
  "password": "temp_password",
  "roles": [
    "superuser",
    "fleet_superuser"
  ]
}

4. Login to Kibana and edit the .fleet-actions-results-ilm-policy ILM policy to include a frozen phase (feel free to modify the rollover conditions and ILM poll interval for testing purposes)

5. Deploy an Elastic Agent with the Osquery Manager integration.

6. Run a few live Osquery queries so that documents are ingested in the .fleet-actions-results data stream

7. Once the initial backing index has rolled over and has entered the frozen phase, check the ILM explain for the same index:

        "ilm": {
          "phase": "frozen",
          "failed_step": "mount-snapshot",
          "phase_definition": "{\"policy\":\".fleet-actions-results-ilm-policy\",\"phase_definition\":{\"min_age\":\"0d\",\"actions\":{\"searchable_snapshot\":{\"snapshot_repository\":\"found-snapshots\",\"force_merge_index\":true}}},\"version\":2,\"modified_date_in_millis\":1733296846017}",
          "snapshot_repository": "found-snapshots",
          "snapshot_name": "2024.12.04-.ds-.fleet-actions-results-2024.12.04-000001-.fleet-actions-results-ilm-policy-6i5yigyat6cymxkb7ub37g",
          "phase_time": "1733297097263",
          "step_info": "{\"type\":\"illegal_argument_exception\",\"reason\":\"requested system indices [.ds-.fleet-actions-results-2024.12.04-000001], but system indices can only be restored as part of a feature state\"}",
          "creation_date": "1733297037388",
          "step_time": "1733297097463",
          "action_time": "1733297038142",
          "is_auto_retryable_error": "true",
          "failed_step_retry_count": "3",
          "snapshot_index_name": ".ds-.fleet-actions-results-2024.12.04-000001",
          "action": "searchable_snapshot",
          "step": "ERROR"
        },

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

@henningandersen @rjernst (since this overlaps many areas) do you know whether we want to allow mounting system indices as a searchable snapshot? I don't think it makes a lot of sense, since we rely on specific types of access to these indices.

[rjernst]

System features (ie the thing snapshotted for system indices/datastreams) are for backing up and restoring. Since they are system state, I'm not sure what the system would look like if a feature were mounted as a searchable snapshot? Users should not be able to search system indices directly, so I can't tell what would be searching them.

[henningandersen]

I think the motivation for allowing this is not clear. System data streams that become very big seems slightly odd, since big data sets would normally be available for querying. As I understand it, allowing to mount backing indices of system data streams is not motivated by a requirement to do so, more that we allow editing the policy too freely.

I'd prefer to perhaps not allow editing the ILM policy of system data streams, or rather they should be controlled through other more specific controls, similar to how DLM works. This could be a specific setting for retention of fleet actions or something different.

So unless there is a compelling reason to be able to mount system indices, I'd prefer not to and then perhaps move towards a more constrained lifecycle setup for system data streams.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)