permission denied when open DB on 2.0.0-beta.0

[WeiBanjo]

What did you do?
Running 2.0.0-beta.0 in Kubernetes

What did you expect to see?
Prometheus starts properly

What did you see instead? Under which circumstances?
Prometheus crashes because of permission denied. I was running v2.0.0-alpha.3 before without problem.

Environment

$ docker info
Containers: 8
Running: 7
Paused: 0
Stopped: 1
Images: 8
Server Version: 1.12.6
Storage Driver: overlay2
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs

• System information:

$ uname -srm
Linux 4.8.6-1.el7.elrepo.x86_64 x86_64

• Prometheus version:

version=2.0.0-beta.0

• Prometheus configuration file:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name:kube-prometheus
spec:
  template:
    metadata:
      labels:
        app: kube-prometheus-v2
        group: kube-prometheus-v2
    spec:
      nodeSelector:
        beta.kubernetes.io/role: prometheus
      containers:
      - name: kube-prometheus
        image: prom/prometheus:v2.0.0-beta.0
        command:
        - /bin/prometheus
        - --storage.tsdb.retention=168h
        - --config.file=/prometheus/prometheus.yaml
        - --storage.tsdb.path=/data
        - --web.enable-admin-api
        - --web.enable-lifecycle
        ports:
        - containerPort: 9090
          hostPort: 9090
          protocol: TCP
          name: prometheus-port
        volumeMounts:
        - mountPath: /data
          name: prometheus-data
        - mountPath: /prometheus
          name: prometheus-home
          readOnly: true
      volumes:
      - hostPath:
          path: /opt/prometheus
        name: prometheus-home
      - hostPath:
          path: /dbvol/prometheus
        name: prometheus-data

• Logs:

time="2017-07-12T19:10:22Z" level=info msg="Starting prometheus (version=2.0.0-beta.0, branch=master, revision=2b5d9159537cbd123219296121e05244e26c0940)" source="main.go:202"
time="2017-07-12T19:10:22Z" level=info msg="Build context (go=go1.8.3, user=root@fc24486243df, date=20170712-12:21:13)" source="main.go:203"
time="2017-07-12T19:10:22Z" level=info msg="Host details (Linux 4.8.6-1.el7.elrepo.x86_64 #1 SMP Mon Oct 31 12:56:11 EDT 2016 x86_64 kube-prometheus-8nxlk (none))" source="main.go:204"
time="2017-07-12T19:10:22Z" level=info msg="Starting tsdb" source="main.go:216"
time="2017-07-12T19:10:22Z" level=error msg="Opening storage failed: open DB in /data: open /data/805465914: permission denied" source="main.go:219"

[fabxc]

As a security improvement, we recently changed default user of our docker container to nobody. The volume you mount into the container must allow that user to read and write files in the data directory. Alternatively, you can change the user running the container.

[gouthamve]

Also, you are using hostPath which comes with the caveat:

the directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged container or modify the file permissions on the host to be able to write to a hostPath volume

So I suggest you add to your template:

securityContext:
  runAsUser: 0

Closing this as this is not a bug in prometheus itself, please reopen if you think otherwise.

[var23rav]

try docker run --rm -it -u $(id -u root):$(id -g root) -p 9091:9090 -v $(pwd)/prom:/prometheus prom_image_name:var23
Notice the use of -u switch -u $(id -u root):$(id -g root) forcing to use the root user in docker.
I got it working in this way. Seems like the volume owner is root.
Since there is no namespace mapping in docker, the folder created in host machine(prom) will be owned by root itself.
The Docker image(prom/prometheus:latest) available in docker hub is no uptodate( 2 months old )

The Dockerfile in master branch has added nobody as a default user. So if you want the update I thinks its better to use prom/promethues:master

[t3hmrman]

Just wanted to add for anyone that might run into this, if you're using Prometheus ephemerally you can use emptyDir (mounting it to /prometheus inside the container) to get through this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.