Enhance get composable template API to include data streams using the template

[dakrone]

When updating a template, it would be helpful to know which data streams might be using the template to know where its update will take effect. We should consider something like:

{
  "index_templates" : [
    {
      "name" : "ilm-history",
      "index_template" : {
        "index_patterns" : ["ilm-history-5*"],
        "template" : {
          "settings" : {
            "index" : {
              "lifecycle" : {
                "name" : "ilm-history-ilm-policy"
              },
              "number_of_shards" : "1",
              "auto_expand_replicas" : "0-1",
              "number_of_replicas" : "0"
            }
          }
        }
        "composed_of" : [ ],
        "priority" : 2147483647,
        "version" : 5,
        "_meta" : {
          "managed" : true,
          "description" : "index template for ILM history indices"
        },
        "data_stream" : {
          "hidden" : true
        },
        "allow_auto_create" : true
      },
      "in_use_by": {
        "data_streams": ["data_stream-1", "my-data"]
      }
    }
  ]
}

See in_use_by

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[yuliacech]

pinging @elastic/kibana-stack-management for visibility

[martijnvg]

Should we also include the use of regular indices via in_use_by object field?

[dakrone]

Should we also include the use of regular indices via in_use_by object field?

I think I would prefer to leave this out for now, as I think it'll cause confusion for users with regard to an (incorrect) assumption that changing a template retroactively changes an index's settings. I also think it'll be kind of confusing with the way that data stream indices are named (in that they wouldn't directly match the index patterns since they are prefixed with things like "partial-" or ".ds-".

[martijnvg]

I think I would prefer to leave this out for now, as I think it'll cause confusion for users with regard to an (incorrect) assumption that changing a template retroactively changes an index's settings.

But that is also true for data streams? Only after a rollover the new template changes are applied in the new write index.
However it isn't tied to a concrete index, so eventually the latest changes to a template is used by a data stream.

Maybe we could return the name of the write alias for none data stream use cases?

I also think it'll be kind of confusing with the way that data stream indices are named (in that they wouldn't directly match the index patterns since they are prefixed with things like "partial-" or ".ds-".

My comment was intended for non data stream usages. I agree returning backing index names is confusing and we shouldn't do that, we should hide backing indices, since the corresponding data stream is returned in data_streams field.

I'm okay with not doing anything for regular indices/write aliases. I was just wondering what we can do for the non data stream use case.