FR: chown the /prometheus for docker containers before start

[hsmade]

When using the docker image for prometheus, you'd want to volume mount /prometheus so that the data can be re-used when restarting the container. Problem is, that docker will create the path on the host as root. When using a scheduler like nomad and using a persistent disk for /prometheus, the same thing happens.

This is a problem as prometheus runs as nobody, and thus doesn't have access to the path that is created by root.

I propose to use a wrapper script as entrypoint that does a chown -R nobody: /prometheus and then starts prometheus as user nobody.

[brancz]

I would argue this is a responsibility of your environment. For example in Kubernetes you can set all of this information using the SecurityContext of a Pod.

[hsmade]

You are right, of course. But then k8s is about the only one, I think (maybe mesos as well?).
It's just a bit cumbersome having this issue with plain docker, and having to wrap this image into another one with entrypoint, etc.

[hairyhenderson]

@hsmade I think the main reason you're seeing this behaviour is because you're bind-mounting a volume. When using a named volume instead (created with docker volume create or some other means), the mount-point will continue to be owned by nobody and prometheus will be able to write to the directory as expected.

For example:

$ docker volume create foo
foo
$ docker run -it --rm --entrypoint=/bin/sh -v foo:/prometheus prom/prometheus:v2.0.0   
/prometheus $ touch hello_world
/prometheus $ ls -alh
total 8
drwxr-xr-x    2 nobody   nogroup     4.0K Nov 19 03:56 .
drwxr-xr-x    1 root     root        4.0K Nov 19 03:56 ..
-rw-r--r--    1 nobody   nogroup        0 Nov 19 03:56 hello_world
/prometheus $ 

[brian-brazil]

Sounds like this is resolved, please let us know if that's not the case.

[jpalczewski]

In

Docker version 18.06.1-ce, build e68fc7a215 
docker-compose version 1.22.0, build unknown
prom/prometheus v2.4.2

It might be again the case: I got

prometheus_1       | level=error ts=2018-09-27T08:39:34.455471782Z caller=main.go:617 err="opening storage failed: lock DB directory: open /prometheus-data/lock: permission denied"   

When I mounted volume from alpine container it seems like it where mounted as root:root.

Applicable fragments from docker-compose:

prometheus: 
    image: prom/prometheus
    command: --config.file=/prometheus-config/prometheus.yml --storage.tsdb.path=/prometheus-data
    volumes: 
      - ./config/prometheus:/prometheus-config
      - prometheus_data:/prometheus-data
(...)
volumes:
  prometheus_data: