Remove terminated pod from summary api.

[Random-Liu]

I found this when debugging a containerd test failure introduced after #77101 (comment).

In some cases, we may have pod stats with the same name and namespace today:

1. The pod sandbox was restarted once, and old pod sandboxes and containers are still kept around;
2. Pods with the same name and namespace are deleted and recreated in a short period of time.

This only happens to the CRI integration, because we return stats for terminated containers from CRI. To keep the behavior of summary api consistent with before, we should filter out those terminated pods.

Or else, this may break metrics which only use labels {container name, pod name, pod namespace} today, e.g. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/prometheus_resource_metrics.go#L47.

Errors like this will be returned when querying metrics:

an error on the server ("An error has occurred while serving metrics:\n\ncollected metric \"kubelet_container_log_filesystem_used_bytes\" { label:<name:\"container\" value:\"test-container-subpath-local-preprovisionedpv-2ws6\" > label:<name:\"namespace\" value:\"provisioning-85\" > label:<name:\"pod\" value:\"pod-subpath-test-local-preprovisionedpv-2ws6\" > gauge:<value:8192 > } was collected before with the same name and label values") has prevented the request from succeeding (get nodes bootstrap-e2e-minion-group-t5wj:10250)

There are 2 metrics affected by this:

• (Added in 1.13) log_metrics: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/collectors/log_metrics.go
• (Added in 1.14) prometheus_resource_metrics: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/prometheus_resource_metrics.go
  So we should cherry-pick this to 1.13 and 1.14 to fix the issue.

/cc @dashpole @davidz627

Signed-off-by: Lantao Liu lantaol@google.com

What type of PR is this?
/kind bug
/kind failing-test

If a pod has a running instance, the stats of its previously terminated instances will not show up in the kubelet summary stats any more for CRI runtimes like containerd and cri-o.

This keeps the behavior consistent with Docker integration, and fixes an issue that some container Prometheus metrics don't work when there are summary stats for multiple instances of the same pod.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Random-Liu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [Random-Liu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feiskyer]

lgtm

/retest

[yujuhong]

Need a release note.

[yujuhong]

Need a release note.
^^^^ still missing :-)

/lgtm

[Random-Liu]

Need a release note.

Done.

[dashpole]

/lgtm

[spiffxp]

This should hopefully address a number of failure clusters that have cropped up in the past week:

• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1#d6ad9db422cbeba079c6
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1#8956e3fb37abab00cd91
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1#ba286364255c7dc0987b
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1#524e0c395a0131c1f7b2

[tedyu]

I wonder why this is needed ?
I assume the func would only keep READY sandboxes.

When I comment out this line, cri_stats_provider_test still passes.