Pods stuck in ContainerCreating state

[ionutleca]

Environmental Info:
RKE2 Version:
v1.20.6+rke2r1 (da4fc2f)

Node(s) CPU architecture, OS, and Version:
Linux server0 4.18.0-193.47.1.el8_2.x86_64 SMP Thu Mar 4 03:03:32 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
When trying to deploy more than ~100 pods we get pods stuck in the ContainerCreating state.
After further digging into the node logs we found that we have results similar to the ones reported here opencontainers/runc#2865

Steps To Reproduce:

• Installed RKE2:

selinux: false
write-kubeconfig-mode: "0644"
etcd-snapshot-retention: 7
etcd-snapshot-schedule-cron: "*/5 * * * *"
tls-san:
- ***
disable:
- rke2-ingress-nginx
kubelet-arg:
- "max-pods=200"

Expected behavior:
Being able to create pods up to the 200 limit

Actual behavior:
Pods are stuck in ContainerCreating well before the 200 limit

Additional context / logs:
Looks like upgrading the runC version to 1.0.0-rc94 from 1.0.0-rc93 (opencontainers/runc#2871) does the trick.
We replaced the runC binary with the new version and the rke2 cluster seemed to successfully schedule pods up to the specified limit.

[Oats87]

@ionutleca how long are you leaving your node with the containers in "ContainerCreating" state? Do these containers ever get past this stage? In our testing with rc93, we have seen nodes hit this issue but end up getting past the backlog with a little bit of time.

[ionutleca]

They were stuck in that state >10 minutes. Since that delay is not acceptable for us, we never waited to see how much they actually stay in that state, considering that we experience no such delays with the fixed runC version.

[cjellick]

@dweomer @brandond @Oats87 will this be addressed by our bump to runc?

[cjellick]

assigning to jacob assuming so

[rajivml]

@Oats87 no they never pass the ContainerCreating state, I have seen them stuck for more than a day

[davidnuzik]

k3s-io/k3s#3305

[rancher-max]

Validated this is working on v1.21.1-alpha5+rke2r1 and v1.20.7-rc1+rke2r1

Pods no longer get stuck in this state. Can increase pod limit successfully via kubelet arg.