How to get Kubernetes log messages from inside Prometheus?

[bboreham]

prometheus/discovery/kubernetes calls Kubernetes code; Kubernetes code logs via glog; there doesn't seem to be a way to set the glog level inside Prometheus.

glog does set up command-line options like -v via flag.Var(), but because nobody is calling flag.Parse() they don't help.

[fabxc]

In the 2.0 branch we are overwriting the error handler and handle the logging of errors from the k8s code base ourselves: https://github.com/prometheus/prometheus/blob/dev-2.0/cmd/prometheus/main.go#L205-L212

[bboreham]

That appears to be yet another thing.

I'm talking about for instance https://github.com/prometheus/prometheus/blob/dev-2.0/vendor/k8s.io/client-go/tools/cache/reflector.go#L236

[fabxc]

I see, ultimately, this is a severe mis-design of Kubernetes libraries in general and we currently don't and cannot provide flags for a basically arbitrarily chosen and unswappable logging libraryies somewhere in our dependencies.

Why are you interested in seeing those debug logs?

[bboreham]

Got an error from discovery; didn't understand it; wanted more context what the k8s library was doing.

More specifically it was a customer running the program so I preferred to say "run with -v=8 and send me the full log" - even though most of the information is useless it is good to get it all in one hit rather than go back 20 times "ok, now try this...".

[fabxc]

Yes, that's right. It would be great to just show those messages when debug logging is enabled but with k8s' current approach I don't see a sane way for us to do it =/

[bboreham]

Request to fix Kubernetes http://github.com/kubernetes/kubernetes/issues/61006