Create system indices on startup if configured

[jaymode]

System indices are necessary for Elasticsearch and the stack to function and should be available as soon as possible. Upon introduction of a new system index or startup of a cluster, a mechanism should be provided so that system indices can be automatically created if configured to do so.

Some system indices that belong to the stack, such as the kibana indices will not have their configuration stored in the system index plugin so this is an example of system indices that we would not want to be created automatically.

For system indices that we know the configuration for ahead of time, then we should enable the automated creation of these to prevent multiple components from needing to maintain their own methods of creating these indices and then later updating them when the time comes. Configuration in this context means the mappings, settings, and any aliases necessary for the system indices.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[pugnascotia]

For system indices that we know the configuration for ahead of time

What would be an example of a system index for which we don't know the configuration ahead of time?

[jaymode]

Kibana is an example since plugins can change some of their mappings.

[pugnascotia]

I spent a while looking for prior art in the codebase, and the closest thing seems to be IndexTemplateRegistry. Most of that class concerns templates and policies, but its cluster state listener implements logic to look for required templates, and loads them if they are missing. We can build something similar for system indices, taking as much care as we can to check that the cluster is in a suitable state.

[jaymode]

There is also a MetadataIndexTemplateService that we should look at as a prior art. I think we can model something after either one to pre-create the system indices. (An aside, at some point I'd like to only have a single implementation for index template installation and upgrades).

[pugnascotia]

It's a shame we don't have a single implementation for creating templates - I have some experimental code to declare which templates a system index relies upon, wait until they exist, and then create the system index. However that doesn't work for e.g. .async-search because that index is created programmatically, including settings and mappings.

[jaymode]

I think we should move away from templates for system indices that we are going to create; the values should be handled in code like async search does. I have been thinking about this some more today and one thought that I had relates to the auto creation of indices; we could defer creating until time of write. We would just hook into the creation of indices so that write requests will create the system index with all of the proper mappings/settings. This has a downside that get and search requests would still need to handle IndexNotFoundException.

@gwbrown @williamrandolph thoughts?

[jaymode]

I'll add a bit more for why it might be best to avoid creating system indices that may go unused by users and that is the fact that each shard has overhead and our docs state 20 shards per GB of heap. If we think about small nodes and clusters, by pre-creating unused indices we are limiting already scarce resources for these users.

[pugnascotia]

That was indeed my concern too about pre-creating indices. So perhaps we can define some common patterns for dealing with possibly-not-existing-yet indices, and creating them on-demand. Then our plugins and 3rd-party plugin authors could just hook into the common framework.

[rudolf]

Just summarising some earlier email discussions:

Since Kibana's indices have dynamic mappings these can't be created on startup. However, auto-created indices still sometimes occur due to user error which then creates a hard to recover from situation.

Kibana would like for the Elasticsearch Kibana plugin to disable auto-create for all it's system indices elastic/kibana#81790

[pugnascotia]

We have now migrated all system indices that Elasticsearch can manage itself to the system indices framework. As such, this issue can now be closed.