Filtering system indices by naming standard

[freesinger]

I tried to export elasticsearch indices and found system index starting with dot like .kibana index appeared. Since it's a convention to name a system index, I set a simple rule to filter all the system indices. Soon I found that users can name an index starting with . and the index can be accepted.

Wondering whether this naming behavior can be prevented by ES engine or can I filter system indices by other metadata of indices?

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[gwbrown]

Hi @freesinger, thanks for using Elasticsearch! We're in the process of implementing some features that I think will fulfill this requirement. I'll give a brief summary below, or you can find our tracking issue at #50251.

First, in the recently-released version 7.7.0, there's a new feature called "hidden indices" using the new index setting index.hidden. These indices are just like normal indices, except that when Elasticsearch expands index patterns (e.g. my-index*), only non-hidden indices will be returned unless hidden indices are requested using the expand_wildcards=all (or =hidden,open) REST parameter. Hidden indices are intended for system data that users may still want to access, such as .watch_history*. Many of the "dot indices" will become hidden indices starting in 7.7.

Second, we're introducing a formalized concept of "system indices", which will be completely locked down and managed by Elasticsearch (or plugins), and access to these indices will only be possible via specialized APIs. This feature is still in development. These formalized system indices are intended for data that is more sensitive or that needs to be tightly controlled - .watches and .security are expected to use this feature in a future release, for example.

Finally, in 7.7.0, we've deprecated the creation of indices with dot-prefixed names that are not 1) hidden indices or 2) managed by ES or a plugin. In a future major version, all dot-prefixed indices will be required to either be hidden or a managed system index.

I think this should cover the use case you're asking about, so I'm going to close this issue. If there's something we've missed and this doesn't cover your use case, please feel free to provide feedback.

If you would like assistance handling dot-prefixed indices in the meantime, I encourage you to ask on our forums.