failed to delete container with error open pidfd: Invalid argument

[babykart]

Hi,

I'm experiencing random stuck on a delete operation of a container with the following error:

Jul 29 11:35:09 lystenia k3s[4952]: E0729 11:35:09.158193    4952 kuberuntime_gc.go:146] Failed to remove container "c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836": rpc error: code = Unknown desc = failed to delete container c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836: `/usr/bin/crun --root /run/runc delete --force c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836` failed: open pidfd: Invalid argument

I only have these errors on Azure nodes:

kubectl get nodes -o wide
NAME       STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
lystenia   Ready    control-plane,master   80d   v1.20.9+k3s1   10.90.1.4     <none>        Ubuntu 18.04.5 LTS   5.4.0-1055-azure   cri-o://1.20.3

I'm using crun static binary downloaded at https://github.com/containers/crun/releases/tag/0.20.1

crun --version
crun version 0.20.1
commit: 38271d1c8d9641a2cdc70acfa3dcb6996d124b3d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

Let me know if you need more information from my environment.

[giuseppe]

pidfd_open seems to fail with EINVAL when the pid is not valid:

	if (pid_has_task(p, PIDTYPE_TGID))
		fd = pidfd_create(p, flags);
	else
		fd = -EINVAL;

Is this issue easy to reproduce? Or did it happen just once?

Could you verify if the container is terminated or in what state the process is?

I am trying to figure out if we need to ignore the issue as the process is already terminated, or if we need to fallback to using kill(2) instead

[giuseppe]

 * This creates a new pid file descriptor with the O_CLOEXEC flag set for
 * the process identified by @pid. Currently, the process identified by
 * @pid must be a thread-group leader. This restriction currently exists
 * for all aspects of pidfds including pidfd creation (CLONE_PIDFD cannot
 * be used with CLONE_THREAD) and pidfd polling (only supports thread group
 * leaders).

Perhaps we should just fallback to kill(2)

[babykart]

Thanks for your quick response.

Is this issue easy to reproduce? Or did it happen just once?

It's difficult to reproduce the stuck of the state terminating of a pod but I will try asap.

Could you verify if the container is terminated or in what state the process is?

I have actually this error message on all deleted containers in the equivalent of the kubelet.log.

For example :

Jul 29 13:23:11 lystenia k3s[4952]: E0729 13:23:11.140546    4952 kuberuntime_gc.go:146] Failed to remove container "c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836": rpc error: code = Unknown desc = failed to delete container c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836: `/usr/bin/crun --root /run/runc delete --force c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836` failed: open pidfd: Invalid argument

For crictl the container in not present :

 crictl ps -a | grep c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836

But the directory is present in /run/runc:

ll /run/runc/c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836/
total 16
drwx------   2 root root    80 Jul 22 20:16 ./
drwx------ 113 root root  2260 Jul 29 09:46 ../
-rwx------   1 root root 11556 Jul 22 20:16 config.json*
-rwx------   1 root root   657 Jul 22 20:16 status*

grep pid /run/runc/c09ad290b6da18adb367f3fd71c41f9cf8fab2c80a76ccaf2ec60bf531137836/status
    "pid": 2327,

And clearly, the process with pid 2327 is not running on the node.

I am trying to figure out if we need to ignore the issue as the process is already terminated, or if we need to fallback to using kill(2) instead

[giuseppe]

thanks, I think the patch could be:

diff --git a/src/libcrun/linux.c b/src/libcrun/linux.c
index 9d246dc..596a712 100644
--- a/src/libcrun/linux.c
+++ b/src/libcrun/linux.c
@@ -3969,6 +3969,8 @@ libcrun_kill_linux (libcrun_container_status_t *status, int signal, libcrun_erro
             return crun_make_error (err, errno, "kill container");
           return 0;
         }
+      if (errno == EINVAL)
+        errno = ESRCH;
       return crun_make_error (err, errno, "open pidfd");
     }

[giuseppe]

PR here: #710