Prometheus docker container shouldn't run as user nobody

[jcollie]

With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99) (#2859). While it's good that Prometheus is no longer running as root, it's not the best idea either to run as "nobody". There are many examples of Docker containers out there that allow changing the UID/GID at runtime that could be borrowed from. At a minimum a different static UID/GID should be used.

[grobie]

it's not the best idea either to run as "nobody"

For what reasons?

[jcollie]

• The nobody user/group was originally designed to be used by NFS to map the UID/GID 0/0 on remote mounts to something else so it's not really a "free" account that can be used by other services, especially ones that own files on disk.
• There's also the fact that the nobody group is not always UID/GID 99 so that UID/GID may be used for other purposes on a system.
• Really the most important reason is that using a fixed UID/GID (especially a low numbered one that is likely to be used for other purposes) makes it impossible to keep data separate from other services. That's why a lot of vendor-supplied Docker containers make it possible to change the UID/GID that the server runs as at runtime.

[grobie]

Sounds good to me. If it's the best practice to give the option to change the UID in order to have control over file owners, wanna send a PR for that?

[nrvnrvn]

It is a curious question.
I've been reviewing security in our docker images recently and while wondering whether I should use a dedicated user or use something from the default set with the least privileges I came across this issue.

So:

• yes, historically there is a lot of legacy stuff that landed in /etc/passwd and others and even in the Linux Standard Base.
• historically, a lot of daemons used to run under the nobody or daemon user.
• but then they discovered that once more than one service starts sharing the same predefined user they find themselves in an unwanted security danger, thus no every service is trying to create and run on behalf of its own user in a shared linux namespace.

BUT

Containers are somewhat different beasts...

See:

• moby/moby#23469 (comment)
• https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf

Basically this instruction does not bind to the specific user but to the specific UID. And you always can easily override it in docker/kubernetes.

$ docker run --user 1234:5678 --rm -ti busybox
/ $ id
uid=1234 gid=5678
/ $ 

So unless you f*cked up somewhere else I find it pretty safe to run a single process container under any uid:gid except root:root, just tell everyone the appropriate uid:gid (not the username:groupname !) to use for the mounted volumes.

Any objections to the above are welcome :)

P.S. From what I've seen so far from various vendors people follow both paths (either creating a separate user or using 65534[debian-based/alpine] or 99[centos-based])

[jcollie]

Kubernetes does not (at least not yet) allow you to change the primary GID of a pod/container. There are some patches making their way through the process but AFAIK they are not part of any released version of Kubernetes. Also, "just" changing the UID/GID that the primary process runs as isn't always sufficient. Depending on the security model, changing the ownership of some files may be necessary as part of the startup process.

In any case, choosing 99 as the default UID/GID isn't the best idea since it's likely that a low number may be in use by other things in the overall system. If we're going to stick with a fixed UID/GID it'd be better to use something semi-random like 9090.

[nrvnrvn]

Let's assume we have two containers running in the system (single or multi-host, no matter), with volumes attached and both of them use the same pair of uid:gids (say 5555:5555). Hence the attached volumes' files are recursively owned by 5555:5555.

What are the possible implications in this case?

[2color]

With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99)

I think the correct UID would be 65534

$ docker run -it --rm --entrypoint sh prom/prometheus:v2.0.0

/prometheus $ id
uid=65534(nobody) gid=65534(nogroup)

UID 99 is the default nobody in Centos

[silentpete]

In our docker compose, I was able to use ":user 1234:5678" concept as mentioned above along with our Ansible deploy play to set the directory permissions on anything volume mounted.

Container runs perfectly

[nemo83]

@silentpete do you have an example of the docker-compose with the user uid/gid configuration? I can set the --user 1234:5678 with docker run, but not with compose.

[silentpete]

Hello @nemo83 ,
I'm currently using the following in my docker compose

version: '2'
services:
  prometheus:
    image: prom/prometheus:v2.2.1
    container_name: prometheus
    hostname: prometheus
    volumes:
      - $PWD/containers/prometheus/:/opt/prometheus/:ro
      # keep data
      - /data/prometheus/:/prometheus/data/:rw
      - /etc/localtime:/etc/localtime:ro
    command:
      - '--config.file=/opt/prometheus/prometheus.yml'
      # prometheus retention period
      - '--storage.tsdb.retention=30d'
    logging:
      driver: "json-file"
    mem_limit: 512m
    memswap_limit: 512m
    restart: unless-stopped
    user: 9090:9090

Can see the user in the container with

# docker exec prometheus ps -ef | grep prometheus
    1 9090       2:56 /bin/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.retention=30d

which I copied out of: https://github.com/silentpete/pg-h.io/blob/master/docker-compose.yml

Hope this helps,
Peter

PS, I created a 9090:9090 (user and group) on my host, then set the volume mounted in directory to have the correct permissions for the data being saved out of the container.

[jsravn]

The main problem w/ running containers as non-root is K8s. How do you guys solve the fact that k8s persistent volumes are generally mounted as root-writeable only? fsGroup is not an option for me either - Openshift, for example, doesn't allow setting fsGroup (to 65543 in this case).

It seems to me that the general trend is to let users specify their UIDs, either via remapping in docker, or runAsUser in k8s. The entire situation is sorta messy unfortunately. I'd suggest sticking with root for maximum compatibility.

[nrvnrvn]

Setting runAsUser and fsGroup to the same values has always worked for me in GKE. It works even with those images where no USER is defined in Dockerfile. Use any UID in range 1-2147483647.

Can't say anything for openshift though. I hope they have got some solution for that anyway.

[jsravn]

Another issue with fsGroup is that it operates at pod level only, so it forces all mounts to that gid which may not be ideal.

Openshift manages UID/GID itself - it has an admission controller I believe which overrides runAsUser / fsGroup. But unfort. it will forbid your pod entirely if you set those values, so building a single pod definition to work across different k8s flavors isn't possible w/ non-root users (as far as I can tell).

[jcollie]

The solution that I've seen/used is to use an init container. Perhaps not ideal but it works.

    spec:
      initContainers:
        - name: change-ownership
          image: busybox
          command:
            - sh
            - -c
            - "chown -Rc 991:991 /data"
          volumeMounts:
            - name: data
              mountPath: /data