Prometheus crashes if it can't write logs to /tmp

[geekofalltrades]

Bug Report

What did you do?

We copy the Prometheus binary into a from-scratch Docker container and run it as a non-root user to reduce our footprint and threat surface. This is a practice we follow with all applications we run in our Kubernetes environment. Recently we upgraded to prometheus v2.4.2 from v2.3.0, where in both cases we were running in containers using this method.

What did you expect to see?

Prometheus should run fine, as v2.3.0 seemed to.

What did you see instead? Under which circumstances?

Prometheus is repeatedly crashing with this error:

W1026 14:54:21.157256       1 reflector.go:341] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:306: watch of *v1.Pod ended with: too old resource version: 19430593 (19430983)
log: exiting because of error: log: cannot create log: open /tmp/prometheus.prometheus-75f465d4d4-hjztt.unknownuser.log.WARNING.20181026-145421.1: no such file or directory

This seems to be a glog error message which occurs when glog can't write a logfile. However, I don't want glog writing to any files at all; it should write all logs to stderr, since we are running in a Docker container and want to delegate log management to Docker. However, I see no way to configure Prometheus or glog to not log to files.

glog apparently has a --logtostderr flag, but it's not being exposed by Prometheus.

Note that the crash is ALWAYS preceded by a warning log from Kubernetes service discovery, so this may be the source of the bug.

This error was also reported on coreos/prometheus-operator here: coreos/prometheus-operator#1932 Notice that the crash in that report is ALSO preceded by a warning from Kubernetes service discovery.

Environment

• System information: no uname in a scratch container.

• Prometheus version:

prometheus, version 2.4.2 (branch: HEAD, revision: c305ffaa092e94e9d2dbbddf8226c4813b1190a0)
  build user:       root@dcde2b74c858
  build date:       20180921-07:22:29
  go version:       go1.10.3

• Logs:

W1026 14:54:21.157256       1 reflector.go:341] github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:306: watch of *v1.Pod ended with: too old resource version: 19430593 (19430983)
log: exiting because of error: log: cannot create log: open /tmp/prometheus.prometheus-75f465d4d4-hjztt.unknownuser.log.WARNING.20181026-145421.1: no such file or directory

[geekofalltrades]

Related issue: kubernetes/kubernetes#61006

[brancz]

It is possible to set the flag programmatically, so we should look into doing that, I think.

[simonpasquier]

Ah, glog 🙄
FWIW I'm working on moving Prometheus to use Go modules and it should be possible then to replace the use of glog by https://github.com/kubermatic/glog-gokit hopefully

[brancz]

Just a painkiller instead of a cure, but it will take a long time until we actually get to remove glog from Kubernetes, so a good option I think @simonpasquier. I'm all for doing that.

[dims]

@brancz fyi, we have klog now in main k/k repo (fork of glog with some stuff fixed)