logstash_internal user unauthorized on custom indices

[pkral3]

The new logstash_internal user in configs cannot write on indices in runtime (v8.0.1):

unauthorized for user [logstash_internal] with roles [logstash_writer] on indices

Solution: In ELK in Stack Management/Users assign to "logstash_internal" user the role as for "elastic" user

[antoineco]

@pkral3 thanks for reporting 👍

May I ask what indices you are writing to? The default role has permissions on the following indices:

• logs-generic-default (the data stream targeted by Logstash's default config)
• logstash-* (the index pattern when data streams are disabled in Logtash's config)
• ecs-logstash-* (the index pattern when data streams are disabled in Logtash's config and ECS schema is enabled)
• logstash (alias used for managing index lifecycle)
• ecs-logstash (alias used for managing index lifecycle when ECS schema is enabled)

[pkral3]

Custom indices that were ok before logstash_internal are not writeable now.
As an example in logstash/pipeline/logstash.conf define any esIndex

filter {
   if [app_id] == "process1" {
      grok {
       match => { "message" => "\[%{TIMESTAMP_ISO8601:logtimestamp}\] \[%{LOGLEVEL:loglevel}%{SPACE}\] \[%{GREEDYDATA:hostname}\] \[%{GREEDYDATA:environment}\] \[%{GREEDYDATA:project}\] \[%{GREEDYDATA:jobname}_%{GREEDYDATA:jobversion}\] \[%{GREEDYDATA:transaction}\] - %{GREEDYDATA:logmessage}" }
      }
      # create an elasticsearch index per month
      mutate { add_field => { "esIndex" => "process1-%{+YYYY.MM}" }}
    }
}

process1 index even existing from before with data in ELK, is not writeable with logstash_internal user (until it has not the role as elastic user has). It worked ok before, nok with 8.0.1 just with update.

edit: formatting (@antoineco)

[antoineco]

@pkral3 OK that's expected because logstash_internal is just assigned a default role during the initial stack startup (logstash_writer), but that role is only suitable for a fairly default log-oriented Logstash pipeline.

Obviously, we can not anticipate all the indices names that end users are going to target.
We can also not open the default permissions too much because that would be quite unsecure, and we don't want to encourage people to use overprivileged users.

I can recommend a few options:

• Create another role via Kibana (e.g. logstash_custom) and assign it the same permissions as the logstash_writer role, just with different indices names, based on your own Logstash config (e.g. process1-*).
• Edit the existing logstash_writer role created by docker-elk, and change/rename the indices patterns it contains.
• Give the logstash_internal user admin extended permissions, like you did, with the risk that this user can be abused to perform more changes than it is supposed to (e.g. perform modifications to the Elasticsearch cluster, read secrets, etc.)

[pkral3]

So there is no solution just to update, build and up -d as before now.

[antoineco]

There is, but only if you don't use custom indices.

Are you suggesting that we make logstash_internal a superadmin like before? That is not very reasonable in terms of security.

The README is quite clear about docker-elk being a template. It is expected that users will use their own pipelines and configurations, and configure their users accordingly. We can't be a one-size fit all.

[antoineco]

Closing because I think the question has been answered.

To sum it up:

• Our default security roles can unfortunately not accommodate every possible custom Logstash pipeline because that would mean giving Logstash access to everything, including potentially sensitive indices.
• We abused the elastic superuser pre-v8, but replacing it with a less privileged user is a step in the right direction.
• It is expected that users manage their security roles based on the indices they use. Besides, the logstash_writer role that ships with docker-elk can be edited freely and checked out into Git, just like the rest of docker-elk's config.