fix OOM killer

[lengrongfu]

What type of PR is this?

/kind bug
/kind failing-test

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes # #119600

Special notes for your reviewer:

The problem with OOM kill looks the same as this pr #116082,

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-containerd-node-e2e-1-7/1683184526936772608/artifacts/tmp-node-e2e-92a26bb7-cos-97-16919-353-1-system.log

Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 kernel: Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efb6ec_09ab_4eec_8ad6_f953eb1a7466.slice/cri-containerd-98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54.scope are going to be killed due to memory.oom.group set
Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time="2023-07-23T18:56:46.329304474Z" level=debug msg="received exit event container_id:\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\" id:\"98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54\" pid:38265 exit_status:137 exited_at:{seconds:1690138606 nanos:325950236}"
Jul 23 18:56:46 tmp-node-e2e-92a26bb7-cos-97-16919-353-1 containerd[910]: time="2023-07-23T18:56:46.424828104Z" level=debug msg="containerEventsChan is full, discarding event {ContainerId:98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54 ContainerEventType:CONTAINER_STOPPED_EVENT CreatedAt:1690138606424696583 PodSandboxStatus:&PodSandboxStatus{Id:17df7d535a8570b5badc642d8f549e6ba8a404cc2c6898a58873a082313678c0,Metadata:&PodSandboxMetadata{Name:oomkill-target-pod,Uid:17efb6ec-09ab-4eec-8ad6-f953eb1a7466,Namespace:oomkiller-test-8312,Attempt:0,},State:SANDBOX_READY,CreatedAt:1690138604987482380,Network:&PodSandboxNetworkStatus{Ip:10.100.0.147,AdditionalIps:[]*PodIP{},},Linux:&LinuxPodSandboxStatus{Namespaces:&Namespace{Options:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,TargetId:,UsernsOptions:nil,},},},Labels:map[string]string{io.kubernetes.pod.name: oomkill-target-pod,io.kubernetes.pod.namespace: oomkiller-test-8312,io.kubernetes.pod.uid: 17efb6ec-09ab-4eec-8ad6-f953eb1a7466,},Annotations:map[string]string{kubernetes.io/config.seen: 2023-07-23T18:56:44.955881675Z,kubernetes.io/config.source: api,},RuntimeHandler:,} ContainersStatuses:[&ContainerStatus{Id:98bac6c4b73cfd4c647647aa80248208b67de059874d4ddfdd1aa43ea3967e54,Metadata:&ContainerMetadata{Name:oomkill-multi-target-container,Attempt:0,},State:CONTAINER_EXITED,CreatedAt:1690138605675227684,StartedAt:1690138605807969879,FinishedAt:1690138606325950236,ExitCode:137,Image:&ImageSpec{Image:registry.k8s.io/e2e-test-images/busybox:1.29-4,Annotations:map[string]string{},},ImageRef:registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937,Reason:Error,Message:,Labels:map[string]string{io.kubernetes.container.name: oomkill-multi-target-container,io.kubernetes.pod.name: oomkill-target-pod,io.kubernetes.pod.namespace: oomkiller-test-8312,io.kubernetes.pod.uid: 17efb6ec-09ab-4eec-8ad6-f953eb1a7466,},Annotations:map[string]string{io.kubernetes.container.hash: 9d865647,io.kubernetes.container.restartCount: 0,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},Mounts:[]*Mount{&Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/17efb6ec-09ab-4eec-8ad6-f953eb1a7466/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,UidMappings:[]*IDMapping{},GidMappings:[]*IDMapping{},},&Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/17efb6ec-09ab-4eec-8ad6-f953eb1a7466/containers/oomkill-multi-target-container/af3cc608,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,UidMappings:[]*IDMapping{},GidMappings:[]*IDMapping{},},},LogPath:/var/log/pods/oomkiller-test-8312_oomkill-target-pod_17efb6ec-09ab-4eec-8ad6-f953eb1a7466/oomkill-multi-target-container/0.log,Resources:&ContainerResources{Linux:&LinuxContainerResources{CpuPeriod:100000,CpuQuota:0,CpuShares:2,MemoryLimitInBytes:15728640,OomScoreAdj:997,CpusetCpus:,CpusetMems:,HugepageLimits:[]*HugepageLimit{},Unified:map[string]string{memory.oom.group: 1,memory.swap.max: 0,},MemorySwapLimitInBytes:15728640,},Windows:nil,},}] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}"

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Sun Jul 30 04:14:48 UTC 2023.

[k8s-ci-robot]

Hi @lengrongfu. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

/ok-to-test
/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 16c8dcef482da9e55d3bd1534e540c414b5d5e7d

[bart0sh]

/triage accepted
/priority important-soon

[tzneal]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, lengrongfu, tzneal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/e2e_node/OWNERS [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tzneal]

Adding f48daf5412edac432c28 so this will show up when searching from k8s-triage.

[pohly]

Still failing?

See https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/119728/pull-kubernetes-node-e2e-containerd/1692065178335580160/

[akhilerm]

@lengrongfu The OOM killer tests are still failing in #120258 . Ref : https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/120258/pull-kubernetes-node-e2e-containerd/1696861389160714240/

I think the issue still exists.

[tzneal]

There is more discussion and a linked containerd issue in #119600. From my testing, containerd is not sending the OOM status sometimes and kubelet is behaving correctly. The initial sleep here doesn't appear to help.