Replace private index settings with index metadata

[rjernst]

Currently we have several "private" index settings. These are index settings which can only be set internally by Elasticsearch, not by the user. One problem with these settings is how/when/if they are copied upon copying an index (eg #44440, although there are other solutions that particular problem).

Last year we discussed using dedicated index metadata for these values, as they are not really settings, but metadata about the index tracked by the system. Such a replacement would need to make typed metadata easier, which is one reason the settings infrastructure is currently used for such values. This issue is a placeholder for discussion/design of such a replacement effort.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Settings)

[jloleysens]

Hi @rjernst , we are currently looking at improvements for Kibana's Upgrade Assistant for 8.0. This change would in particular help us address elastic/kibana#40564.

I would like to understand the implications for Upgrade Assistant a bit better; will any special action be required when handling older indices for re-indexing?

[rjernst]

@jloleysens This issue describes a path we would like to take with settings for greater simplification to users, but there is not yet a concrete plan for when this would happen. To answer your question, though, when this work is completed these will no longer be settings. No external action would be necessary.

[jaymode]

I looked into this in the context of system indices. Custom index metadata was modified to be more acceptable in #32749 and then used by CCR in #33546. This means that code running within elasticsearch has a method to put a Map of metadata within the IndexMetadata in the cluster state.

In terms of what is missing, it looks like it is the typing of the metadata. I do wonder if this is an issue though since the metadata is serialized by a plugin or other caller and not updateable via an API as far as I can see.

[gwbrown]

I'm labeling this team-discuss to determine whether we're likely to make an effort to do this in the near future or if there's another approach we should take here. #44440 is related.

[gwbrown]

We discussed this in the core/infra sync today and decided that while we like this proposal from an aesthetic perspective, it would be a good bit of effort and there are areas where we can spend that development effort more effectively in the near future.

Therefore, we're going close this issue and #44440 and promote the idea in #44440 (comment) to its own issue.