Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flood_stage not working with x-pack security #33119

Closed
nachogiljaldo opened this issue Aug 24, 2018 · 8 comments

Comments

Projects
None yet
7 participants
@nachogiljaldo
Copy link

commented Aug 24, 2018

Elasticsearch version (bin/elasticsearch --version): 6.3.2 (but the same symptoms appear with 6.0.1 and 6.4.0, although I did not debug them).

Plugins installed: [discovery-file, repository-s3]

JVM version (java -version):

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux 4efa9a510327 4.4.0-66-generic #87~14.04.1-Ubuntu SMP Fri Mar 3 17:32:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
cluster.routing.allocation.disk.watermark.flood_stage does not take effect when x-pack security is installed.
The logs show a message like:

[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only

but settings are not applied to the indices.

Steps to reproduce:

  1. Create a cluster with x-pack security
  2. Make sure cluster.routing.allocation.disk.threshold_enabled is enabled and configure cluster.routing.allocation.disk.watermark.flood_stage
  3. You should see on the logs something like:
[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only
  1. I would expect that my indices become read-only, but it does not happen

Provide logs (if relevant):
Did some debugging and it appears this exception is thrown:

ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [_system]]
    at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:575)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:553)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:158)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:183)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:161)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$23(AuthenticationService.java:432)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:441)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$16(AuthenticationService.java:333)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:340)

and after some more debugging, it seems the culprit is https://github.com/elastic/elasticsearch/blob/v6.3.2/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/SystemPrivilege.java#L17-L27 which apparently should include (at the very least) permissions for indices:admin/settings/update or even indices:admin/settings/* (provided it does raise any security concerns).

FWIW, I tried that (by compiling that class in 6.3.2 and substituting it) and it worked as expected after it

@tvernum tvernum changed the title flood_stage knot working with x-pack security flood_stage not working with x-pack security Aug 24, 2018

@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Aug 24, 2018

@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Aug 24, 2018

@bleskes

This comment has been minimized.

Copy link
Member

commented Sep 9, 2018

@tvernum @jaymode this looks important to me. I bumped into it while catching up. Was this discussed?

@albertzaharovits

This comment has been minimized.

Copy link
Contributor

commented Sep 9, 2018

Not sure how these "internal" actions should be handled but I can pick this up, if nobody else is bidding for it.

@tvernum

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

@bleskes Sorry, no it dropped off the radar, but I agree it's important.

I think there are three changes we should make (in order from highest priority)

  1. Change the _system role to grant the appropriate privilege
  2. DiskThresholdMonitor.markIndicesReadOnly should have some sort or error handling/logging if the index settings change fails.
  3. The AuthorizationService should have some logging level (DEBUG/TRACE) which logs all failures. IT should be possible to turn up the logging levels see these errors without needing to hack the code.
@jaymode

This comment has been minimized.

Copy link
Member

commented Sep 10, 2018

I would be interested in an assert on access denieds for the system user to try to be more proactive about catching these than relying on logging

@bleskes

This comment has been minimized.

Copy link
Member

commented Sep 23, 2018

@tvernum @jaymode +1 on my end. Are you guys going to work on it?

@tvernum

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2018

I'll make sure someone picks this up in time for 6.5.0

@bizybot bizybot self-assigned this Sep 25, 2018

bizybot added a commit to bizybot/elasticsearch that referenced this issue Sep 25, 2018

[Authz] Allow update settings action for system user
When the `cluster.routing.allocation.disk.watermark.flood_stage` watermark
is breached, DiskThresholdMonitor marks the indices as read only. This
failed when x-pack security was present as System user does not have
privilege for update settings action("indices:admin/settings/update").
This commit adds the required privilege for System user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes elastic#33119

bizybot added a commit to bizybot/elasticsearch that referenced this issue Sep 25, 2018

[Authz] Allow update settings action for system user
When the `cluster.routing.allocation.disk.watermark.flood_stage` watermark
is breached, DiskThresholdMonitor marks the indices as read only. This
failed when x-pack security was present as System user does not have
privilege for update settings action("indices:admin/settings/update").
This commit adds the required privilege for System user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes elastic#33119

bizybot added a commit that referenced this issue Oct 4, 2018

[Authz] Allow update settings action for system user (#34030)
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119

bizybot added a commit that referenced this issue Oct 4, 2018

[Authz] Allow update settings action for system user (#34030)
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119

kcm added a commit that referenced this issue Oct 30, 2018

[Authz] Allow update settings action for system user (#34030)
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.