Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flood_stage not working with x-pack security #33119

Closed
nachogiljaldo opened this issue Aug 24, 2018 · 8 comments
Closed

flood_stage not working with x-pack security #33119

nachogiljaldo opened this issue Aug 24, 2018 · 8 comments
Assignees
@bizybot
Labels
>bug :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC

Comments

@nachogiljaldo
Copy link

Elasticsearch version (bin/elasticsearch --version): 6.3.2 (but the same symptoms appear with 6.0.1 and 6.4.0, although I did not debug them).

Plugins installed: [discovery-file, repository-s3]

JVM version (java -version):

java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux 4efa9a510327 4.4.0-66-generic #87~14.04.1-Ubuntu SMP Fri Mar 3 17:32:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
cluster.routing.allocation.disk.watermark.flood_stage does not take effect when x-pack security is installed.
The logs show a message like:

[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only

but settings are not applied to the indices.

Steps to reproduce:

  1. Create a cluster with x-pack security
  2. Make sure cluster.routing.allocation.disk.threshold_enabled is enabled and configure cluster.routing.allocation.disk.watermark.flood_stage
  3. You should see on the logs something like:
[2018-08-24T08:07:51,796][WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] flood stage disk watermark [97%] exceeded on [1hFDDw9yRxGCIlfVRKuCCg][instance-0000000000][/app/data/nodes/0] free: 173.5mb[1.1%], all indices on this node will be marked read-only
  1. I would expect that my indices become read-only, but it does not happen

Provide logs (if relevant):
Did some debugging and it appears this exception is thrown:

ElasticsearchSecurityException[action [indices:admin/settings/update] is unauthorized for user [_system]]
    at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:575)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:553)
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:158)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:173)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:167)
    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:149)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:183)
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:161)
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$23(AuthenticationService.java:432)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:441)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$16(AuthenticationService.java:333)
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:340)

and after some more debugging, it seems the culprit is https://github.com/elastic/elasticsearch/blob/v6.3.2/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/SystemPrivilege.java#L17-L27 which apparently should include (at the very least) permissions for indices:admin/settings/update or even indices:admin/settings/* (provided it does raise any security concerns).

FWIW, I tried that (by compiling that class in 6.3.2 and substituting it) and it worked as expected after it
@tvernum tvernum changed the title flood_stage knot working with x-pack security flood_stage not working with x-pack security Aug 24, 2018
@tvernum tvernum added >bug :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC labels Aug 24, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@bleskes
Copy link
Contributor

bleskes commented Sep 9, 2018

@tvernum @jaymode this looks important to me. I bumped into it while catching up. Was this discussed?

@albertzaharovits
Copy link
Contributor

Not sure how these "internal" actions should be handled but I can pick this up, if nobody else is bidding for it.

@tvernum
Copy link
Contributor

tvernum commented Sep 10, 2018

@bleskes Sorry, no it dropped off the radar, but I agree it's important.

I think there are three changes we should make (in order from highest priority)

  1. Change the _system role to grant the appropriate privilege
  2. DiskThresholdMonitor.markIndicesReadOnly should have some sort or error handling/logging if the index settings change fails.
  3. The AuthorizationService should have some logging level (DEBUG/TRACE) which logs all failures. IT should be possible to turn up the logging levels see these errors without needing to hack the code.

@jaymode
Copy link
Member

jaymode commented Sep 10, 2018

I would be interested in an assert on access denieds for the system user to try to be more proactive about catching these than relying on logging

@bleskes
Copy link
Contributor

bleskes commented Sep 23, 2018

@tvernum @jaymode +1 on my end. Are you guys going to work on it?

@tvernum
Copy link
Contributor

tvernum commented Sep 23, 2018

I'll make sure someone picks this up in time for 6.5.0

@bizybot bizybot self-assigned this Sep 25, 2018
bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Sep 25, 2018
[Authz] Allow update settings action for system user
3dffdab 
When the `cluster.routing.allocation.disk.watermark.flood_stage` watermark
is breached, DiskThresholdMonitor marks the indices as read only. This
failed when x-pack security was present as System user does not have
privilege for update settings action("indices:admin/settings/update").
This commit adds the required privilege for System user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes elastic#33119
bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Sep 25, 2018
[Authz] Allow update settings action for system user
5c31415 
When the `cluster.routing.allocation.disk.watermark.flood_stage` watermark
is breached, DiskThresholdMonitor marks the indices as read only. This
failed when x-pack security was present as System user does not have
privilege for update settings action("indices:admin/settings/update").
This commit adds the required privilege for System user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes elastic#33119
@bizybot bizybot mentioned this issue Sep 25, 2018
Merged
bizybot added a commit that referenced this issue Oct 4, 2018
@bizybot
[Authz] Allow update settings action for system user (#34030)
81227dc 
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119
bizybot added a commit that referenced this issue Oct 4, 2018
@bizybot
[Authz] Allow update settings action for system user (#34030)
22cb973 
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119
kcm pushed a commit that referenced this issue Oct 30, 2018
@bizybot @kcm
[Authz] Allow update settings action for system user (#34030)
80860ee 
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119
@olfuerniss olfuerniss mentioned this issue Aug 29, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bizybot bizybot

Labels
>bug :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@bizybot @nachogiljaldo @bleskes @tvernum @jaymode @albertzaharovits @elasticmachine