Metric kube_pod_container_status_terminated_reason don't detect all events

[lombardialess]

What happened:
The metric kube_pod_container_status_terminated_reason is still experiment but since a very long time.
This metric can be very useful for monitoring alerts but is not detecting all "Errors" or "OOMKilled" often the event is not collected.

What you expected to happen:
Trace of any event of termination

How to reproduce it (as minimally and precisely as possible):
I don't think is reproducible but for example OOMKilled is very often not detected in the metric

Anything else we need to know?:

Environment:

• kube-state-metrics version: 2.9.2
• Kubernetes version (use kubectl version): 1.25.6
• Cloud provider or hardware configuration: Azure AKS
• Other info:

[dashpole]

cc @CatherineF-dev
/triage accepted
/assign @dgrisonnet

[dobesv]

Have you tried kube_pod_container_status_last_terminated_reason ? It might be that the pod was restarted and the state was moved to the "last" field.

[lombardialess]

yes but on this metric I cannot set an alert, if the pod goes OOM the last reason is always OOMKilled and last field always 1 also if this happen multiple times.
Is better to have a metric saved on kube_pod_container_status_terminated_reason that can be triggered with an alert all the times that the issue happen.

[TadeoCloud]

Same problem here. When an OOMKill is detected, kube-metric always shows the value "1". But no more OOMKills have occurred.

And if I test the function increase, it detects a change from null to zero.

[TadeoCloud]

When using kube-metrics v2.4.1 the sum value goes from "null" to "1" and then back to "null".

[dobesv]

It would be nice to find a good alert query for this.

Maybe when querying the kube_pod_container_status_last_terminated_reason you also need to query whether the pod has restarted recently?

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} and on(namespace, pod, container) (increase(kube_pod_container_status_restarts_total[30m]) > 0)

[TadeoCloud]

It would be nice to find a good alert query for this.

Maybe when querying the kube_pod_container_status_last_terminated_reason you also need to query whether the pod has restarted recently?

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} and on(namespace, pod, container) (increase(kube_pod_container_status_restarts_total[30m]) > 0)

It seems to be working! I'm testing it. Thanks

[CatherineF-dev]

QQ: is it working?

If you don't have other questions, we will close it.

[TadeoCloud]

Confirmed, it's working for me. Thanks!

[dgrisonnet]

FWIW, there is also a new metrics in kubelet to better detect OOMKilled containers: kubernetes/kubernetes#108004

Closing since the initial problem seem to have been resolved.