K8S Pods with gVisor are stuck in running state with load of zombie children

[MrBatschner]

Description

We have a Kubernetes cluster on which we are observing the following behaviour for quite some time:

$ k get po
NAME                                                              READY   STATUS            RESTARTS   AGE
4db95b31-ba4c-4de9-904e-391f7a0df5f5-1759791593-driver-0 1/1     Running    0         1d12h      2025-10-06T23:03:16Z     ip-172-30-4-239.eu-central-1.compute.internal

That Pod claims to be running but it should have terminated quite some time ago. That Pod is running in a gVisor sandbox and contains a huge amount of zombie children. The Pods sandbox id is ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff.

When we get on to the node, this is the process tree starting with the shim:

root     3999813  0.0  0.0 1236552 11352 ?       Sl   Oct06   0:07 /var/bin/containerruntimes/containerd-shim-runsc-v1 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd
root     3999846  0.0  0.0 1273180 26988 ?       Ssl  Oct06   0:02  \_ runsc-gofer --log-format=json --panic-log=/var/log/pods/c9ecbc0f-4819-4868-b337-c17e3e4dc9eb_4db95b31-ba4c-4de9-904e-391f7a0df5f5-1759791593-driver-0_b6935fdf-7909-4c24-9a4e-3f25567a0681/gvisor_pan
root     3999850  9.2  0.0 3596852 71420 ?       Ssl  Oct06 320:22  \_ runsc-sandbox --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff/log.json --log-format=json --panic-log=/var/log/pods/c9ecbc
root     3999909  0.0  0.0  16456     0 ?        Ss   Oct06   0:00  |   \_ [exe]
root     3999953  0.0  0.0  17992     0 ?        S    Oct06   0:00  |   \_ [exe]
root     3999954  0.0  0.0  17992     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000220  0.0  0.0  16672    28 ?        S    Oct06   0:00  |   \_ [exe]
root     4000221  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001010  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001011  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001016  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001017  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001018  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001029  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001030  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001031  0.0  0.0  16672    28 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000543  0.0  0.0  16624   564 ?        S    Oct06   0:00  |   \_ [exe]
root     4000544  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000551  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000552  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000553  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000566  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000567  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000601  0.0  0.0  16624   564 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4000545  0.0  0.0  16480   768 ?        S    Oct06   0:00  |   \_ [exe]
root     4000546  0.0  0.0  16480   768 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001552  0.0  0.0  16480     0 ?        S    Oct06   0:00  |   \_ [exe]
root     4001553  0.0  0.0  16480     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001554  0.0  0.0  16480     0 ?        S    Oct06   0:00  |   \_ [exe]
root     4001556  0.0  0.0  16480     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001555  0.0  0.0  16936     0 ?        S    Oct06   0:00  |   \_ [exe]
root     4001557  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001585  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001586  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001587  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001588  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001589  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001595  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001596  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001597  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001599  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001600  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001601  0.0  0.0  16936     0 ?        SN   Oct06   0:01  |   |   \_ [exe]
root     4001608  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001617  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001619  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001621  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001622  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001623  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001624  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001625  0.0  0.0  16936     0 ?        SN   Oct06   0:00  |   |   \_ [exe]
root     4001559  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001647  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001648  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001649  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001651  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001652  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001653  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001654  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001659  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001678  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001679  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001758  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001759  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001760  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001761  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001762  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001763  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001764  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001765  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001766  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001767  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001768  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001769  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001770  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001771  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001772  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001773  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001774  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001775  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001776  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001783  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001784  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001785  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001786  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001787  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001788  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001789  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001790  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001791  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001793  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001794  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001795  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     4001796  0.0  0.0      0     0 ?        ZN   Oct06   0:00  |   \_ [exe] <defunct>
root     3999955  0.0  0.0 1272028 22180 ?       Sl   Oct06   0:00  \_ runsc --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff/log.json --log-format=json --pa

If we would like to query runsc to list all containers, it blocks forever:

$ root@ip-172-30-4-239:/var/bin/containerruntimes#./runsc --root /run/containerd/runsc/k8s.io list

Having a look with strace about where it blocks:

root@ip-172-30-4-239:/var/bin/containerruntimes# strace ./runsc --root /run/containerd/runsc/k8s.io list
[...]
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x3178b20, FUTEX_WAIT_PRIVATE, 0, NULL

It blocks at acquiring a futex. But not always, sometimes, it blocks when it tries to connect to the problematic sandbox' socket:

root@ip-172-30-4-239:/var/bin/containerruntimes# strace ./runsc --root /run/containerd/runsc/k8s.io list
[...]
socket(AF_UNIX, SOCK_STREAM, 0)         = 7
connect(7, {sa_family=AF_UNIX, sun_path="/run/containerd/runsc/k8s.io/runsc-ae0b66c097179b61e39a73400d574b4c19812d527d2e2ad9c4dfe92f767246ff.sock"}, 107

With this, it is also impossible to run any runsc debug command, they would all block either when acquiring the futex or connecting to the sandbox socket.

Any hints on how we can track this down further?

Steps to reproduce

We are not able to provide exact steps to reproduce the problem. It appears on only one of our many clusters but there on several different nodes. Also, the problem appeared across many different gVisor versions we tried throughout the last year.

runsc version

runsc version release-20250820.0
spec: 1.2.0

docker version (if using docker)

-

uname

Linux ip-172-30-4-239.eu-central-1.compute.internal 6.6.87-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.6.87-0gl0~bp1592 (2025-04-17) x86_64 GNU/Linux

kubectl (if using Kubernetes)

$ k version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.32.7

$ k get node ip-172-30-4-239.eu-central-1.compute.internal
NAME                                            STATUS   ROLES    AGE   VERSION
ip-172-30-4-239.eu-central-1.compute.internal   Ready    <none>   12d   v1.31.7

repo state (if built from source)

No response

runsc debug logs (if available)

[nixprime]

Can you check if there are any OOM kill messages in dmesg?

Can you try adding:

[runsc_config]
  panic-signal = "12"

to /run/containerd/runsc/config.toml on your nodes (which causes the containerd shim to pass -panic-signal=12 to runsc), then send the runsc-sandbox process SIGUSR2 (signal 12) when it gets into this state to force a panic and dump stack traces?

[MrBatschner]

Thank you for your reply.

dmesg had no mentions of an OOM kill and in fact no other traces that could give a clue about the problem - it was as clean as it could get.

We will try and set the panic-signal in our runsc config - unfortunately, the way we run our clusters and workloads, this might take a while but we will be getting back with the results.