Prometheus docker container shouldn't run as user nobody

[jcollie]

With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99) (#2859). While it's good that Prometheus is no longer running as root, it's not the best idea either to run as "nobody". There are many examples of Docker containers out there that allow changing the UID/GID at runtime that could be borrowed from. At a minimum a different static UID/GID should be used.

[grobie]

it's not the best idea either to run as "nobody"

For what reasons?

[jcollie]

• The nobody user/group was originally designed to be used by NFS to map the UID/GID 0/0 on remote mounts to something else so it's not really a "free" account that can be used by other services, especially ones that own files on disk.
• There's also the fact that the nobody group is not always UID/GID 99 so that UID/GID may be used for other purposes on a system.
• Really the most important reason is that using a fixed UID/GID (especially a low numbered one that is likely to be used for other purposes) makes it impossible to keep data separate from other services. That's why a lot of vendor-supplied Docker containers make it possible to change the UID/GID that the server runs as at runtime.

[grobie]

Sounds good to me. If it's the best practice to give the option to change the UID in order to have control over file owners, wanna send a PR for that?

[nrvnrvn]

It is a curious question.
I've been reviewing security in our docker images recently and while wondering whether I should use a dedicated user or use something from the default set with the least privileges I came across this issue.

So:

• yes, historically there is a lot of legacy stuff that landed in /etc/passwd and others and even in the Linux Standard Base.
• historically, a lot of daemons used to run under the nobody or daemon user.
• but then they discovered that once more than one service starts sharing the same predefined user they find themselves in an unwanted security danger, thus no every service is trying to create and run on behalf of its own user in a shared linux namespace.

BUT

Containers are somewhat different beasts...

See:

• docker run with "--user nobody" which should have no ability to login system, however ,entered system with "-it" option moby/moby#23469 (comment)
• https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf

Basically this instruction does not bind to the specific user but to the specific UID. And you always can easily override it in docker/kubernetes.

$ docker run --user 1234:5678 --rm -ti busybox
/ $ id
uid=1234 gid=5678
/ $ 

So unless you f*cked up somewhere else I find it pretty safe to run a single process container under any uid:gid except root:root, just tell everyone the appropriate uid:gid (not the username:groupname !) to use for the mounted volumes.

Any objections to the above are welcome :)

P.S. From what I've seen so far from various vendors people follow both paths (either creating a separate user or using 65534[debian-based/alpine] or 99[centos-based])

[jcollie]

Kubernetes does not (at least not yet) allow you to change the primary GID of a pod/container. There are some patches making their way through the process but AFAIK they are not part of any released version of Kubernetes. Also, "just" changing the UID/GID that the primary process runs as isn't always sufficient. Depending on the security model, changing the ownership of some files may be necessary as part of the startup process.

In any case, choosing 99 as the default UID/GID isn't the best idea since it's likely that a low number may be in use by other things in the overall system. If we're going to stick with a fixed UID/GID it'd be better to use something semi-random like 9090.

[nrvnrvn]

Let's assume we have two containers running in the system (single or multi-host, no matter), with volumes attached and both of them use the same pair of uid:gids (say 5555:5555). Hence the attached volumes' files are recursively owned by 5555:5555.

What are the possible implications in this case?

[2color]

With the update to 2.0, the Docker container was modified so that Prometheus runs as the user "nobody" (UID/GID 99/99)

I think the correct UID would be 65534

$ docker run -it --rm --entrypoint sh prom/prometheus:v2.0.0

/prometheus $ id
uid=65534(nobody) gid=65534(nogroup)

UID 99 is the default nobody in Centos

[silentpete]

In our docker compose, I was able to use ":user 1234:5678" concept as mentioned above along with our Ansible deploy play to set the directory permissions on anything volume mounted.

Container runs perfectly

[nemo83]

@silentpete do you have an example of the docker-compose with the user uid/gid configuration? I can set the --user 1234:5678 with docker run, but not with compose.

[silentpete]

Hello @nemo83 ,
I'm currently using the following in my docker compose

version: '2'
services:
  prometheus:
    image: prom/prometheus:v2.2.1
    container_name: prometheus
    hostname: prometheus
    volumes:
      - $PWD/containers/prometheus/:/opt/prometheus/:ro
      # keep data
      - /data/prometheus/:/prometheus/data/:rw
      - /etc/localtime:/etc/localtime:ro
    command:
      - '--config.file=/opt/prometheus/prometheus.yml'
      # prometheus retention period
      - '--storage.tsdb.retention=30d'
    logging:
      driver: "json-file"
    mem_limit: 512m
    memswap_limit: 512m
    restart: unless-stopped
    user: 9090:9090

Can see the user in the container with

# docker exec prometheus ps -ef | grep prometheus
    1 9090       2:56 /bin/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.retention=30d

which I copied out of: https://github.com/silentpete/pg-h.io/blob/master/docker-compose.yml

Hope this helps,
Peter

PS, I created a 9090:9090 (user and group) on my host, then set the volume mounted in directory to have the correct permissions for the data being saved out of the container.

[jsravn]

The main problem w/ running containers as non-root is K8s. How do you guys solve the fact that k8s persistent volumes are generally mounted as root-writeable only? fsGroup is not an option for me either - Openshift, for example, doesn't allow setting fsGroup (to 65543 in this case).

It seems to me that the general trend is to let users specify their UIDs, either via remapping in docker, or runAsUser in k8s. The entire situation is sorta messy unfortunately. I'd suggest sticking with root for maximum compatibility.

[nrvnrvn]

Setting runAsUser and fsGroup to the same values has always worked for me in GKE. It works even with those images where no USER is defined in Dockerfile. Use any UID in range 1-2147483647.

Can't say anything for openshift though. I hope they have got some solution for that anyway.

[jsravn]

Another issue with fsGroup is that it operates at pod level only, so it forces all mounts to that gid which may not be ideal.

Openshift manages UID/GID itself - it has an admission controller I believe which overrides runAsUser / fsGroup. But unfort. it will forbid your pod entirely if you set those values, so building a single pod definition to work across different k8s flavors isn't possible w/ non-root users (as far as I can tell).

[jcollie]

The solution that I've seen/used is to use an init container. Perhaps not ideal but it works.

    spec:
      initContainers:
        - name: change-ownership
          image: busybox
          command:
            - sh
            - -c
            - "chown -Rc 991:991 /data"
          volumeMounts:
            - name: data
              mountPath: /data

[Raffaello]

it's not the best idea either to run as "nobody"

For what reasons?

I have this example:

the mounted config prometheus.yml file must have a xx4(-o+r) permission to be read, otherwise would be permission denied, as the file would be mounted as a root user.

So i guess a little bit elaborated entrypoint could help, preparing the required files permission, taking advantages of gosu or similar as also explained here: stack-overflow-link
not sure if it is possible with busybox anyway.

[the-maldridge]

I was just bit by this, Nomad doesn't have a way to set up the bind users like this so I am having to set up a bind volume manually, kind of defeating the point of using the container.

[ohritz]

Ran into this issue this week when trying to run prometheus within OpenShift, was testing things out with open-telemetry and wanted to run prometheus as a deployment with the quay.io/prometheus/prometheus:v2.43.0 image.

Setting a user with securityContext is not permitted, at least not anything lower than uid: 1001030000
And the init container solution does not work either.

As per OpenShift Guidlines https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines under the section Support Arbitrary User IDs

All that would be required to not get permission denied is to add

RUN chgrp -R 0 /prometheus && \
    chmod -R g+rwX /prometheus

I built my own image with

FROM quay.io/prometheus/prometheus:v2.43.0

USER 0
RUN chgrp -R 0 /prometheus && \
    chmod -R g+rwX /prometheus

USER nobody

and this worked.

Would it be a alright to add it to the Dockerfile, I can make a PR if needed.

[marvinnitz18]

@ohritz please open a merge request for that, i would support it

[marvinnitz18]

i created one with your suggestions: #12728

[ohritz]

@marvinnitz18 I'm sorry I never got to making a PR, I honestly forgot about it. I see you made you took over, thanks!

[marvinnitz18]

@ohritz no worries, lets see if it can get merged

[elysweyr]

Hi,
for those of you encountering this issue with the official prometheus image and need a fix right away, I got you covered.

Red Hat itself is publishing a patched version of the prometheus image which is suitable for rootless execution on OpenShift.
Image: registry.redhat.io/openshift4/ose-prometheus

Here's a link to their latest image at the time this comment was published.

On lines 26-28 you'll find their applied patch.

RUN mkdir -p /prometheus && \
    chgrp -R 0 /etc/prometheus /prometheus && \
    chmod -R g=u /etc/prometheus /prometheus

[beorn7]

Hello from the bug scrub. #12728 is currently under review, so hopefully this will be addressed soon.

[naanlizard]

Perhaps we should rephrase this ticket as (and update the MR to) "Support docker --user flag" or something along those lines

The exact user the dockerfile builds/runs as doesn't matter if we can set e.g. --user 1000:1000 when running it