Expand beats_system role privileges #40876

ycombinator · 2019-04-04T19:47:49Z

Traditionally we have recommended that Beats send their monitoring data to the production Elasticsearch cluster. Beats do this by calling the POST _monitoring/bulk API. When Security is enabled this API call requires the cluster:admin/xpack/monitoring/bulk privilege. The built-in beats_system role has this privilege.

Going forward, Beats will be able to send their monitoring data directly to the monitoring Elasticsearch cluster. Beats will do this by calling the regular POST _bulk API. When Security is enabled this API call requires the indices:data/write/bulk privilege. Further, the call has to be able to create any indices that don't exist.

This PR expands the built-in beats_system role's privileges. Specifically, it adds index-level write and create_index privileges for .monitoring-beats-* indices.

This will allow Beats users to continue using the beats_system role for the new direct monitoring route when Security is enabled.

elasticmachine · 2019-04-04T19:47:51Z

Pinging @elastic/es-security

elasticmachine · 2019-04-04T19:47:53Z

Pinging @elastic/es-core-features

urso · 2019-04-05T13:12:48Z

They way monitoring works and is configured in Beats, this change makes sense. It creates the least user-disruption.

What about the template/index mapping? Is it setup correctly if Beats uses the bulk API?

ycombinator · 2019-04-05T19:48:31Z

What about the template/index mapping? Is it setup correctly if Beats uses the bulk API?

The monitoring plugin in Elasticsarch sets up the monitoring index templates. So as long as the cluster that Beats is shipping to has xpack.monitoring.enabled: true (which is the default), the correct monitoring index templates will get created when the cluster is created.

tvernum · 2019-04-15T04:40:39Z

...core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java

+                        new String[] { "monitor", MonitoringBulkAction.NAME},
+                        new RoleDescriptor.IndicesPrivileges[]{
+                            RoleDescriptor.IndicesPrivileges.builder()
+                                .indices(".monitoring-beats-*").privileges("create_index", "write").build()


Can we make this create instead of write ?
That gives the ability to index documents (including overwriting existing documents), but not call the update or delete APIs.

I believe that's sufficient for your purposes, and would get us closer to having least privilege users & roles.

Thanks, indeed create instead of write is sufficient here as the .monitoring-beats-* indices are meant to be append-only and the create privilege allows for bulk indexing as well.

Fixed in 6bf4d8b.

tvernum · 2019-04-15T04:42:02Z

...src/test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java

                is(false));
+        assertThat(beatsSystemRole.indices().allowedIndicesMatcher(CreateIndexAction.NAME).test(index), is(true));
+        assertThat(beatsSystemRole.indices().allowedIndicesMatcher(IndexAction.NAME).test(index), is(true));
+        assertThat(beatsSystemRole.indices().allowedIndicesMatcher(DeleteAction.NAME).test(index), is(true));


If you're happy with my suggestion above, then this will need to be false.

…ta to monitoring cluster

ycombinator · 2019-04-16T01:05:27Z

@tvernum I made the change you suggested. This is ready for your review again. Thanks!

tvernum

LGTM

Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege. [Going forward](elastic/beats#9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist. This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.

jpountz · 2019-07-05T10:17:10Z

@ycombinator This change is on 7.3 already so I'm assuming there is nothing left to backport and removed the backport pending label.

ycombinator added >enhancement backport pending :Data Management/Monitoring :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC v8.0.0 v7.2.0 labels Apr 4, 2019

ycombinator requested a review from urso April 4, 2019 19:48

ycombinator mentioned this pull request Apr 5, 2019

Update Beats monitoring docs re: direct shipping to monitoring cluster elastic/beats#11637

Closed

tvernum requested changes Apr 15, 2019

View reviewed changes

ycombinator force-pushed the security/beats-system-role-expand branch from 348390b to 6bf4d8b Compare April 15, 2019 12:23

ycombinator added 5 commits April 15, 2019 10:26

Fixing variable name copy pasta

5283c49

Expanding beats_system role so Beats can directly index monitoring da…

4051f0f

…ta to monitoring cluster

Fixing line length

6f9622b

Reducing privileges

e1e4cce

Fixing typo from fat fingering

d9e4cb0

ycombinator force-pushed the security/beats-system-role-expand branch from 811b163 to d9e4cb0 Compare April 15, 2019 17:27

tvernum approved these changes Apr 16, 2019

View reviewed changes

ycombinator merged commit 64768bf into elastic:master Apr 16, 2019

ycombinator deleted the security/beats-system-role-expand branch April 16, 2019 01:25

ycombinator mentioned this pull request Apr 16, 2019

[7.x] Expand beats_system role privileges (#40876) #41232

Merged

jakelandis added v7.3.0 and removed v7.2.0 labels Jun 17, 2019

jpountz removed backport pending :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC labels Jul 5, 2019

axw mentioned this pull request Sep 13, 2019

APM Server monitoring using apm_system role fails elastic/apm-server#2708

Closed

This was referenced Sep 30, 2019

Apm system monitoring 7x #47302

Closed

APM system_user #47668

Merged

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand beats_system role privileges #40876

Expand beats_system role privileges #40876

ycombinator commented Apr 4, 2019 •

edited

Loading

elasticmachine commented Apr 4, 2019

elasticmachine commented Apr 4, 2019

urso commented Apr 5, 2019

ycombinator commented Apr 5, 2019

tvernum Apr 15, 2019

ycombinator Apr 15, 2019

tvernum Apr 15, 2019

ycombinator commented Apr 16, 2019

tvernum left a comment

jpountz commented Jul 5, 2019

Expand beats_system role privileges #40876

Expand beats_system role privileges #40876

Conversation

ycombinator commented Apr 4, 2019 • edited Loading

elasticmachine commented Apr 4, 2019

elasticmachine commented Apr 4, 2019

urso commented Apr 5, 2019

ycombinator commented Apr 5, 2019

tvernum Apr 15, 2019

Choose a reason for hiding this comment

ycombinator Apr 15, 2019

Choose a reason for hiding this comment

tvernum Apr 15, 2019

Choose a reason for hiding this comment

ycombinator commented Apr 16, 2019

tvernum left a comment

Choose a reason for hiding this comment

jpountz commented Jul 5, 2019

ycombinator commented Apr 4, 2019 •

edited

Loading