Adding more labels to kube_pod_status_phase

[rajatjindal]

we had a situation where # of failed pod counts increased dramatically, and we were wondering what happened.

on debugging we found 2 nodes were having docker issues and most of the failed nodes were being scheduled on those problematic nodes.

i think it will be useful to add more labels to kube_pod_status_phase, so that we can run query like all failed pods count group by node.

[rajatjindal]

I will be more than happy to open a PR if we agree that this is a reasonable ask.

[brancz]

That can be done at query time. Prometheus supports joins, so you can join pod info on the phase to figure out the node.

[andyxning]

Close this via @brancz's comments above.

In case you have not used the join syntax of Prometheus, there is an example in the #137 comment with some query like:

sum by(node)(avg by(node,pod,namespace)(kube_pod_info{}) * on(pod, namespace) group_right(node) kube_pod_status_phase{phase="Failed"})

[rajatjindal]

thank you very much guys. I will try this out tonight