Index templates are not compatible with file-based component templates

[jeanfabrice]

Elasticsearch Version

8.16.1

Installed Plugins

none

Java Version

bundled

OS Version

Kubernetes 1.30.4 on Ubuntu 22.04

Problem Description

Elasticsearch returns HTTP/400 when creating an index template via API call where one of the associated component templates is a file-based component template

This, for example, prevents integration from upgrading when it ships with an updated index templates.
E.g:

(I'm sorry I can't copy-paste the error as the message is not persistent when I try to move my mouse to select it.)

Additional context in these Slack threads:

• https://elastic.slack.com/archives/CJZ9U5G9X/p1732801746634399
• https://elastic.slack.com/archives/C0D8ST60Y/p1680259171010499

Steps to Reproduce

1. Create and apply test@custom component template via ECK stackconfigpolicy

apiVersion: stackconfigpolicy.k8s.elastic.co/v1alpha1
kind: StackConfigPolicy
metadata:
  name: elasticsearch
spec:
  elasticsearch:
    indexTemplates:
      componentTemplates:
        test@custom:
          template:
            settings:
              refresh_interval: "10s"

2. Wait for the stackconfigpolicy to be in ready phase (change propagated)

$ k get stackconfigpolicies.stackconfigpolicy.k8s.elastic.co elasticsearch 
NAME            READY   PHASE   AGE
elasticsearch   2/2     Ready   1d

3. Go to Kibana developer tools and create a test index template this way

PUT _index_template/test
{
  "index_patterns": ["test*"],
  "template": {
    "settings": {},
    "mappings": {}
  },
  "priority": 200,
  "composed_of": ["test@custom","test2@custom"],
  "ignore_missing_component_templates": ["test@custom","test2@custom"]
}

4. See the error returned

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": """Failed to process request [PutTemplateV2Request[name=test, cause=api, create=false, index_template={"index_patterns":["test*"],"template":{"settings":{}},"composed_of":["test@custom","test2@custom"],"priority":200,"ignore_missing_component_templates":["test@custom","test2@custom"]}]] with errors: [[component_template:test@custom] is reserved by [file_settings]]"""
      }
    ],
    "type": "illegal_argument_exception",
    "reason": """Failed to process request [PutTemplateV2Request[name=test, cause=api, create=false, index_template={"index_patterns":["test*"],"template":{"settings":{}},"composed_of":["test@custom","test2@custom"],"priority":200,"ignore_missing_component_templates":["test@custom","test2@custom"]}]] with errors: [[component_template:test@custom] is reserved by [file_settings]]"""
  },
  "status": 400
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

The problem is that we intentionally do not allow mixing file-settings templates with API templates. So you cannot have a component template in a file setting (test@custom) that is used by an API template (test). That is explicitly checked for and prevented in the code. The reason is that it could leave the system in an inconsistent state if the file were deleted.

However, in this case (and in virtually all cases where the component template is a @custom one), the component template is listed in ignore_missing_component_templates, so I think there is really no harm in it being deleted (although that probably needs more discussion among the team).

In the meantime, the only workaround I can think of is to delete the file setting template and to either create a test@custom component template through the API, or modify the test template directly (not recommended since they would lose those changes on upgrade).

[jeanfabrice]

Hey @masseyke, thank you so much for looking at this.

Since ECK stackconfigpolicy has support for component templates, one could expect being able to define his own @Custom component templates via Infrastructure as Code oriented pattern, rather than API or UI. Shouldn't the current limitation be documented, on the ECK stackconfigpolicy side maybe?

[masseyke]

I added a PR for that -- elastic/cloud-on-k8s#8457.

[Magister-Machinis]

Ran headlong into this error and google brought me to this issue. I'd like to keep the index templates related to logstash sourced data associated with logstash and have a collection of component templates for common mappings and such across the system. With the StackConfigPolicy kind as it is now it very quickly gets large enough to be difficult to review and update, nevermind troubleshoot.
If the mixing of api and file setting templates were permitted with a warning message generated explaining the risk, this could sufficiently inform users what risks they are accepting. Alternatively, allowing multiple StackConfigPolicy instances to be applied to kibana/elasticsearch would allow it to be broken up into more manageable and mono-purposed yml files, alleviating the "giant blob of text" syndrome that the current design suffers from.