Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

red* (asterisk) index health #52304

Open
cjcenizal opened this issue Feb 13, 2020 · 9 comments
Open

red* (asterisk) index health #52304

cjcenizal opened this issue Feb 13, 2020 · 9 comments
Assignees

Comments

@cjcenizal
Copy link
Contributor

@cjcenizal cjcenizal commented Feb 13, 2020

Is this a bug on master? Cat indices reports an index as having red* health. This seems like a recent change because our client-side logic in Kibana expects health to be either green, yellow, or red.

image

@jasontedor

This comment has been minimized.

Copy link
Member

@jasontedor jasontedor commented Feb 13, 2020

I suspect this is related to #51456 which converted the ILM history index to a hidden index, from #50452. Hidden indices are not expanded, which means they shouldn't resolve in health/stats calls. When we can't get this information, we mark an index as red*, behavior that's been there since the beginning (092fd6f).

@gwbrown @jaymode Can you think about the proper handling of this?

@cjcenizal cjcenizal added the :ES-UI label Feb 13, 2020
@elasticmachine

This comment has been minimized.

Copy link
Collaborator

@elasticmachine elasticmachine commented Feb 13, 2020

Pinging @elastic/es-ui (:ES-UI)

@gwbrown gwbrown self-assigned this Feb 13, 2020
@gwbrown

This comment has been minimized.

Copy link
Contributor

@gwbrown gwbrown commented Feb 13, 2020

Still digging into this, but some notes:

  • This reproduces only with Security enabled, hidden indices do not show up in the list at all with Security disabled.
  • This reproduces only when the hidden index in question has an alias.
  • I believe this is due to how aliases are handled in IndicesAndAliasResolver when resolving "all indices".
  • This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why.
  • However, hidden indices do not show up in the ClusterHealthResponse retrieved to build the _cat/indices response, which is the proximate cause of the red* health value.

@jaymode, this also caused me to realize that there's currently no way to view hidden indices in the _cat APIs (except for this bug), as the IndicesOptions used for them are hardcoded. We'll need to add a new request parameter specific to the _cat APIs, or always include hidden indices in the _cat APIs, or rework the _cat APIs to accept IndicesOptions (which I haven't thought through the consequences of but seems like a significant change). Do you have an opinion on which way would be better?

@elasticmachine

This comment has been minimized.

Copy link
Collaborator

@elasticmachine elasticmachine commented Feb 14, 2020

Pinging @elastic/es-core-infra (:Core/Infra/Core)

@tvernum

This comment has been minimized.

Copy link
Contributor

@tvernum tvernum commented Feb 14, 2020

This reproduces only when the hidden index in question has an alias.

It sounds like yet another consequence of #32238 (and the connected issues).
The current behaviour around Security's expansion of wildcards doesn't work well with aliases.

My preference (but there's a variety of different views across the ES team) is actually to move wildcard resolution from being in a pre-action interceptor to being in a pluggable core component.

Right now security tries to replace wildcards with fixed names before the action runs. That means we replace a* with every index and alias that you have access to that starts with a. By the time the core action runs it doesn't have wildcards, it has real names.
But many actions have different handling for what they should do if an alias is explicitly included in the request, vs what they should do if it is covered by a wildcard. So when security is enabled you get the former behaviour, but without security you get the latter. (As I write this, I guess the same issue now applies to expanding wildcards to hidden index names).

If we move wildcard expansion into some pluggable component, then the actions always deal with wildcards, but security would be able to take responsibility for filtering that expansion to only include indices that the user has access to.

This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why

I think this is the same situation as above (per my "I guess..."). Good luck with that one @jaymode - I guess you'll never really get to escape this probem :)

This reproduces only when the hidden index in question has an alias

What semantics do we intend to have when there is an alias to a hidden index?
Is the alias also hidden? It sounds like that's what you're aiming for, but the whole sceario seems weird to me.

@jaymode

This comment has been minimized.

Copy link
Member

@jaymode jaymode commented Feb 14, 2020

Do you have an opinion on which way would be better?

I knew that the cat APIs had hardcoded their indices options from when I went through all of the REST APIs to see where we needed to update for hidden indices. My thought at the time was that hidden indices could be left out of the cat apis and I vaguely remember that this was discussed during the last EAH. Currently we have code like

final IndicesOptions indicesOptions = IndicesOptions.strictExpand();

My suggestion is to simply modify this to:

final IndicesOptions indicesOptions = IndicesOptions.fromRequest(request, IndicesOptions.strictExpand());

The current behaviour around Security's expansion of wildcards doesn't work well with aliases.

This is true and has always been problematic with security. Ultimately part of the problem is the feature where actions can be authorized against an alias without the user having permissions against the underlying index.

This causes hidden indices with aliases show up in GET */_settings (which is also a bug), although I don't yet understand the logic as to why.

It is the same issue where security doesn't introspect anything about an alias when doing expansion

AliasOrIndex aliasOrIndex = metaData.getAliasAndIndexLookup().get(index);
if (aliasOrIndex.isAlias()) {
//it's an alias, ignore expandWildcardsOpen and expandWildcardsClosed.
//complicated to support those options with aliases pointing to multiple indices...
//TODO investigate supporting expandWildcards option for aliases too, like es core does.
return indicesOptions.ignoreAliases() == false;
}

@jaymode

This comment has been minimized.

Copy link
Member

@jaymode jaymode commented Feb 14, 2020

What semantics do we intend to have when there is an alias to a hidden index?
Is the alias also hidden? It sounds like that's what you're aiming for, but the whole sceario seems weird to me.

In some cases, the alias should also be hidden in my opinion. The reason why I think that should be the case is that we have ILM, which write aliases are a part of that and if we think about some of our hidden indices, I think ILM makes sense to be used for those. So if expanding wildcards and an alias points to only hidden indices, then the alias should also be hidden. When hidden and non-hidden are mixed in an alias, that's when we start getting into the cases where it gets muddy. My suggestion:

  1. By default an alias is visible
  2. If an alias only points to hidden indices, it is also hidden.
  3. If an alias points to a mix of hidden and non-hidden indices, it is visible.
  4. Allow setting a property on an alias to enable it to be hidden ("is_hidden" : true)

Item 4 is the ugliest aspect especially when it comes to the API for managing aliases and I think we could get away with just doing items 1-3.

@rjernst

This comment has been minimized.

Copy link
Member

@rjernst rjernst commented Feb 14, 2020

Since aliases are supposed to act like indices on their own, why does what they point to matter? Could we decouple them so that they can point to any mix of hidden and non-hidden indices, and only the hidden property on the alias matters for visibility? This would match with the dot prefix behavior in linux filesystems we have modeled hidden indices on, where an alias is like a symlink.

@jaymode

This comment has been minimized.

Copy link
Member

@jaymode jaymode commented Feb 14, 2020

why does what they point to matter?

I dropped a sentence in my reply while editing; I was using the write index support and attempting to align this with that. In that case, there is the implicit write index on an alias when the alias only points to a single index. Hidden and write indices are different so they aren't going to align exactly, so we can avoid that implicit support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.