kube-state-metrics keeps failed pods around for kube_pod_* metrics

[dlespiau]

Some kube_pod_* metrics exist for Failed pods:

kube_pod_info
kube_pod_owner
kube_pod_labels
kube_pod_created
kube_pod_start_time

kube_status_phase shows that kube-state-metrics is populating this group of metrics with information about the failed (non running) pods:

kubectl is indeed confirming there's only one pod running (and so are the kube_pod_container_* metrics).

I would expect the metrics above to only show Pending|Running pods, maybe Unknown but I'm not sure when the last value is set and seems like it could be a transient state more than anything else.

[andyxning]

@dlespiau This is the desired behavior for kube-state-metrics to scrape kube_pod_info or other metrics for failed pods. As stated many times in other places by @brancz , kube-state-metrics just scrape the object from Kubernetes and emit the corresponding metrics via different criteria.

If you do not like two show kube_pod_info or other metrics for failed pods, you can join kube_status_phase with other metrics to only show up running or pending pods info.

On the other hand, some users may still want to query those info to or analyze or add an alarm.

[dlespiau]

Hi Andy. That makes sense, but your "As stated many times in other places" makes me think that's not what people are expecting the default behaviour to be, ie. by default report only running pods and join with kube_status_phase for error conditions. That said, that'd be an "API" breaking change, so that certainly seems like a deal breaker.

Maybe having it somewhere at a prominent place in the docs would alleviate the problem of an unexpected behaviour, it seems like a big trap people fall into.

[andyxning]

@dlespiau IMHO, it should be a common sense that metrics like kube_pod_info should contain all the object of that type. So, i do not know where we should add this statement and how to say about this truth. WDYT @brancz

[brancz]

I think it would be fair to add a note high in then readme calling out that kube-state-metrics is about representing the state of the Kubernetes API as metrics as opposed to “what kubectl shows”.

[andyxning]

@brancz Will you help to add this tip?

[dlespiau]

Thank you!