[k8scluster] add k8s.container.status_last_terminated_reason metric

[povilasv]

Component(s)

receiver/k8scluster

Is your feature request related to a problem? Please describe.

I would like to get some container state metrics, about termination. One use case is to know whether the container was terminated due to OOM kill or application errorr.

Example happening in pod:

kubectl get pod X -o yaml

...
apiVersion: v1
kind: Pod
...
status:
  containerStatuses:
  - containerID: containerd://07ff7db2706d20ddd26c6257c1c1dbf176917f855c7feef0e05db88159b1584d
    image: image
    imageID:  imageId
    lastState:
      terminated:
        containerID: containerid
        exitCode: 2
        finishedAt: "2024-02-05T23:17:38Z"
        reason: Error
        startedAt: "2024-02-05T23:17:33Z"

Kube State Metrics has this modelled as this Prometheus metric:

kube_pod_container_status_last_terminated_reason
labels:
container=
pod=
namespace=
reason=
uid=

Ref: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/pod-metrics.md

So would be great to have a similar metric.

Describe the solution you'd like

Not sure how to model it in otel correctly, but I'm thinking something like this:

  k8s.container.status_last_terminated_reason:
    enabled: false
    description: Last terminated reason of container. The unit is always 1.
    unit: ""
    attributes:
      - reason
    gauge:
      value_type: int

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/k8scluster: @dmitryax @TylerHelmuth @povilasv. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@povilasv reason can be any value right? Does k8s have a specific set of reasons? This metric feels similar to Phase, should we try to follow the same pattern?

[povilasv]

I've grepped thru the Kubernetes code base and couldn't find a set of possible Reason values. It seems that it's set dynamically somehow, and really hard to find the actual possible values.

I think this is why Kube-State-Metrics also did this similiar way -> https://github.com/kubernetes/kube-state-metrics/blob/122e5e899943eb78eaf3e366733d5dbec6613ac0/internal/store/pod.go#L339

[TylerHelmuth]

In that case an attribute makes sense to me

[povilasv]

Opened a PR for it #31281 :)

[avanish-vaghela]

Would adding a restart_reason label to the k8s.container.restart metric make sense?

[TylerHelmuth]

@avanish-vaghela maybe. Please open another issue for that request.

[dmitryax]

I don't get why another metric is needed for this. Can it be an optional resource attribute instead?

[avanish-vaghela]

@dmitryax I believe you meant metric attribute as the attribute's value would change based on the state when metric value is registered

[povilasv]

I don't get why another metric is needed for this. Can it be an optional resource attribute instead?

Thanks for feedback. I think you are right this should be resource attribute. I think im still used modelling everything as metric. Given Otel has Resource model and this seems to fit more . Container is the resource and status_last_terminated_reason is the attribute of it.

[atoulme]

Can this issue be closed now that #31505 is merged?

[ElfoLiNk]

@povilasv, thank you for enabling us to detect OOMKilled errors. However, which metric should we utilize for detecting CrashLoopBackOff?

kube-state-metrics is utilizing kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}.

Thank you.

[povilasv]

@ElfoLiNk you need a different resource attribute (status_waiting_reason) for this. I suggest you file a new issue.

[ElfoLiNk]

@ElfoLiNk you need a different resource attribute (status_waiting_reason) for this. I suggest you file a new issue.

Created #32457