Implement hidden indices

[jaymode]

This change introduces a new feature for indices so that they can be
hidden from wildcard expansion. The feature is referred to as hidden
indices. An index can be marked hidden through the use of an index
setting, index.hidden, at creation time. One primary use case for
this feature is to have a construct that fits indices that are created
by the stack that contain data used for display to the user and/or
intended for querying by the user. The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

Hidden indices have the following properties:

• API calls for all indices (empty indices array, _all, or *) will not
  return hidden indices by default.
• Wildcard expansion will not return hidden indices by default unless
  the wildcard pattern begins with a .. This behavior is similar to
  shell expansion of wildcards.
• REST API calls can enable the expansion of wildcards to hidden
  indices with the expand_wildcards parameter. To expand wildcards
  to hidden indices, use the value hidden in conjunction with open
  and/or closed.
• Creation of a hidden index will ignore global index templates. A
  global index template is one with a match-all pattern.
• Index templates can make an index hidden, with the exception of a
  global index template.
• Accessing a hidden index directly requires no additional parameters.

This PR is a draft/wip of the hidden indices feature described in #50251. There is still a good amount of work left to do (below), but opening this is primarily to provide the opportunity for early feedback.

To Do:

• Clean up changes in IndexNameExpressionResolver
• Global templates do not apply to hidden indices
• Separate removal of IndicesOption#fromByte and associated test to its own PR Remove unused IndicesOptions#fromByte method #50665
• Open issue for wildcard states parsing TODO so we do not lose track of it IndicesOptions wilcard state parsing is too lenient #50762
• Integration with the security IndicesAndAliasesResolver
• More tests (primarily unit tests)
• Documentation
• Double check uses of IndicesOptions in various API calls to see if looking at hidden by default makes sense (for example, the ClusterHealthRequest since if a hidden index isn't healthy then the cluster isn't healthy)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[gwbrown]

Looks good overall, not much to comment on aside from stuff already on the to-do list.

[ycombinator]

The desire to keep them hidden is
to avoid confusing users when searching all of the data they have
indexed and getting results returned from indices created by the
system.

@jaymode This makes sense when it comes to searching user data, but how about monitoring data like index stats? Specifically, does it make sense for the GET /stats API call to return stats for all indices as it used to before hidden indices were introduced?

ref: #56949 and elastic/beats#18639
cc: @chrisronline

[ycombinator]

Also somewhat related to my previous comment, GET _cat/indices respects the expand_wildcards parameter but GET _cat/shards does not. The latter shows all indices' shards, as if expand_wildcards=all were set. Further, if expand_wildcards is indeed specified in a GET _cat/shards request it barfs out a 400 response.

[jaymode]

@ycombinator thanks for the input here. I went back and looked at the original design which only stated that hidden indices would be hidden during wildcard resolution for search; they should not be hidden in non data APIs. I'm going to discuss this with a few others to ensure we still want this behavior.

Further, if expand_wildcards is indeed specified in a GET _cat/shards request it barfs out a 400 response.

Not every cat API supports the wildcard expansion parameters and I think that should be a separate issue.

[ycombinator]

I went back and looked at the original design which only stated that hidden indices would be hidden during wildcard resolution for search; they should not be hidden in non data APIs. I'm going to discuss this with a few others to ensure we still want this behavior.

@jaymode Any updates on this discussion? Should GET /stats return stats for hidden indices as well?

Not every cat API supports the wildcard expansion parameters and I think that should be a separate issue.

++ will move this to a separate issue.

[jaymode]

@ycombinator we discussed this today in the system indices sync and did not come away with a clear conclusion to make any changes. The consensus was that wildcard resolution differences between APIs could be confusing. A stats call returning hidden indices while a search call doesn't feels wrong. That said we did say we could like at maybe changing classes of APIs and how they behave but we need a strong argument.

[ycombinator]

Thanks for discussing this @jaymode. Acknowledged that the GET _stats call will continue to not return hidden indices by default.