[elastic-agent] Evaluate whether root is the correct user to run the elastic-agent docker image as

[andrewvc]

There's currently some confusion about which user to run dockerized `elastic-agent containers as. This has popped up in elastic/cloud-on-k8s#4794

We currently do create an elastic-agent user in the Dockerfile for agent, but since the docker agent isn't documented we have no established best practice here.

We also have two beats in conflict, Heartbeat does not allow script based monitors to run as the root user. This has created confusion for users who do want to run agent as a root user as in elastic/cloud-on-k8s#4794 . It seems that the main rationale for root is that metricbeat requires root for hostpath volumes, which are recommended against for security reasons. There may be other reasons for root I'm not aware.

So, we need to resolve these problems:

1. We need clear advice, either we tell users to run as root or not. If we do prefer root there's no need for the elastic-agent user.
2. We need to ensure that the choose made in 1. is secure. Since container root is not a 'real' root, AFAICT there's not a huge risk in running as UID 0 in a docker container. Esp. if the container daemon is set to run as a regular user (which is the best defense).
3. If running as non-root we have issues with setcap, as covered in [elastic-agent][heartbeat] Heartbeat binary should have setcap privs for ICMP ping #27651

Would appreciate thoughts from @ruflin @blakerouse and others.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[joshbressers]

It is considered a best practice to not run as root. Google has a nice blog explaining this
https://cloud.google.com/architecture/best-practices-for-operating-containers#avoid_running_as_root

[andrewvc]

@joshbressers makes sense, +1 on running as non-root.

I've also opened this companion issue to cover the issues we have using linux capabilities in agent: #27651

[blakerouse]

I think we need to ensure that running as non-root allows access to all the same contents the Elastic Agent needs. If the hostfs is bind mounted into the container and its not running as root will it still be able to collect all the needed system logs and metrics? How will this work once we can run Endpoint Security in the container? Will Elastic Agent be able to spawn that not being root? Can we still inject eBPF without being root?

There is a lot of open questions that I think we need product to be involved into before we can just drop root.

[andrewvc]

According to this relatively this LWN article only operations on sockets can be done in an unprivileged manner.

That said, there's some middle ground here. We could run the agent as root, but specify else where which user each beat runs as then run the binary as the less privileged elastic-agent user when we execve to beats that don't need that privilege.

[joshbressers]

@blakerouse ACK, it's a journey that often takes a long time.

FWIW, there is a CAP_BPF capability. I admit I've never used it though

[andrewvc]

More detail on CAP_BPF is available here . Does this satisfy the needs of metricbeat @blakerouse ?

[blakerouse]

We currently do not have support for running Endpoint in the container at the moment. It's just ensuring that we do not switch to one way to then need to switch it back later. That is why I think it's important to get product management involved so that we don't rush to a decision and change it, without knowing the future vision of the project and what permissions we will need.

I would prefer that Elastic Agent does not run as root in a container as well, but we need to know the side-effects of that as of today and in the future. Elastic Agent does run as root on an installed host, and that cannot change (because of Endpoint). So keeping them at the same level in containers is also another idea instead of having 2 levels of permissions (1 for installed on a host vs inside of a container).

The CAP_BPF might work in the future for Endpoint once its starts protection of container systems like Kubernetes, but again that is yet to be known.

As for metricbeat now, I think we would need to just try it and see if running as not root with a bind mounted hostfs will work. That is something that I do not know enough about to give enough context to give a conclusion.

[ruflin]

In general I like the idea of not having to run as root. If I remember correctly, the root part is likely coming from the early days of Beats and we took it over to Elastic Agent. My biggest concern is that we don't know yet in detail what will break and if anything will break without having root and we need to investigate this before making any decision. @andrewkroh You might have some more insights here?

[andrewvc]

Closing in favor of elastic/elastic-agent#147 which has more detail