Update Settings API with _all target should not attempt to override settings on system indices

[ChrisHegarty]

Elasticsearch Version

8.3.2, 8.5-SNAPSHOT

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin chegar-MBP.local 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101 arm64

Problem Description

When following the instructions to delay allocation when a node leaves, on a node that has been upgraded from 7.17.6 to 8.3.2 (or later), the settings update fails with:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]"
  },
  "status": 500
}

Steps to Reproduce

1. Create a cluster 7.17.6.
2. Add the Kibana sample data.
3. Create a transform using the sample data.
4. Upgrade to 8.3.2 (or 8.5-SNAPSHOT)
5. Run the update index settings API with the superuser role and observe the error.

PUT /_all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "15m"
  }
}

Logs (if relevant)

The logs show:

[2022-08-29T13:57:30,231][WARN ][o.e.a.a.i.s.p.TransportUpdateSettingsAction] [chegar-MBP.local] Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]
[2022-08-29T13:57:30,247][WARN ][r.suppressed             ] [chegar-MBP.local] path: /_all/_settings, params: {index=_all}java.lang.IllegalStateException: Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]
  at org.elasticsearch.server@8.5.0-SNAPSHOT/org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.masterOperation(TransportUpdateSettingsAction.java:110)
  at org.elasticsearch.server@8.5.0-SNAPSHOT/org.elasticsearch.action.admin.indices.settings.put.TransportUpdateSettingsAction.masterOperation(TransportUpdateSettingsAction.java:44)
  at org.elasticsearch.server@8.5.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:128)
  at org.elasticsearch.server@8.5.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:279)
  at org.elasticsearch.server@8.5.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)

Workaround

Use the following regex as the target path /*,-.*/_settings (rather than _all). This will ensure that hidden indices are not selected.

[ChrisHegarty]

This is a bug - the update index settings API, with the default expand_wildcards query value of open, should not match system indices.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[thecoop]

Are there any situations where _all should change settings on system indices?

[ChrisHegarty]

@gwbrown @williamrandolph

Are there any situations where _all should change settings on system indices?

[thecoop]

I've been unable to reproduce this on 8.3.2 or latest 8.5.0, the settings command completes successfully with the data & transforms in place.

[grcevski]

@thecoop, perhaps the missing step is getting 7.17 going first, upgrading to 8.3 and then trying?

[thecoop]

I added some sample data from kibana to a running 7.17.6, added a random transform, then copied the data directory into a 8.5 snapshot directory, started it up, then ran the update index command

[gwbrown]

Are there any situations where _all should change settings on system indices?

If you explicitly set expand_wildcards=open,hidden (or all). I don't expect this to be used a lot, but I could see it being useful for things like e.g. clearing write blocks.

[stefnestor]

This bug still surfaces 8.6.x. Is there any potential resolution ETA?

E.g. trying

PUT _all/_settings?expand_wildcards=open
{"settings": {"index.unassigned.node_left.delayed_timeout": "5m"} }

errors

{
  "error": {
    "reason": "Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]",
    "root_cause": [
      {
        "reason": "Cannot override settings on system indices: [.transform-internal-*] -> [index.unassigned.node_left.delayed_timeout]",
        "type": "illegal_state_exception"
      }
    ],
    "type": "illegal_state_exception"
  },
  "status": 500
}

[yquirion]

I'm also having that issue. I've talked with support and they cannot tell me when this will be fixed. The only thing we can do is to "vote" to have it fixed, but I doubt that will change anything...

The only way to make the change is the workaround provided by @ChrisHegarty .

PUT *,-.*/_settings?expand_wildcards=open,hidden
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}