-
Notifications
You must be signed in to change notification settings - Fork 1.6k
containerd with cgroup v2 fails to collect metrics in network=host mode #12723
Copy link
Copy link
Open
Labels
type: bugSomething isn't workingSomething isn't working
Description
Description
Below is the configuration of runsc.toml
root@node:/root # cat /etc/containerd/runsc.toml
[runsc_config]
network = "host"
systemd-cgroup = "true"I found that kubectl top pod did not collect metrics for the relevant pods. Checking with journalctl, the following logs were discovered:
Mar 12 08:22:53 node containerd[1776702]: time="2026-03-12T08:22:53.319444658Z" level=error msg="collecting metrics for e8e1c7ea0ccd6e1d075d0954546e1e1b70b010648d8789c6aab8581e876108ed" error=""runsc" did not terminate successfully: : unknown"
Then try to execute
strace -f -s 100000 runsc --root=/run/containerd/runsc/k8s.io events -stats e8e1c7ea0ccd6e1d075d0954546e1e1b70b010648d8789c6aab8581e876108ed
revealed the logs:
[pid 1862679] <... nanosleep resumed>NULL) = 0
[pid 1862678] <... recvmsg resumed>{msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\"success\":false,\"err\":\"/proc/net/dev is not opened for hostinet\",\"result\":null}", iov_len=1535}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_TRUNC|MSG_DONTWAIT) = 79
[pid 1862678] futex(0xc00006c958, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1862679] getpid( <unfinished ...>
[pid 1862678] <... futex resumed>) = 1
[pid 1862680] <... futex resumed>) = 0
[pid 1862678] shutdown(7, SHUT_RDWR <unfinished ...>
[pid 1862679] <... getpid resumed>) = 1862678
[pid 1862678] <... shutdown resumed>) = 0
[pid 1862680] futex(0xc00006c958, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 1862678] write(8, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 1862679] tgkill(1862678, 1862678, SIGURG <unfinished ...>
[pid 1862678] close(8) = 0
[pid 1862678] close(7 <unfinished ...>
[pid 1862679] <... tgkill resumed>) = 0
[pid 1862678] <... close resumed>) = 0
[pid 1862678] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1862678, si_uid=0} ---
[pid 1862679] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 1862678] rt_sigreturn({mask=[]}) = 0
[pid 1862678] exit_group(128 <unfinished ...>
[pid 1862679] <... nanosleep resumed>NULL) = -1 (errno 18446744073709551385)
[pid 1862678] <... exit_group resumed>) = ?
[pid 1862682] <... futex resumed>) = ?
[pid 1862681] <... futex resumed>) = ?
[pid 1862680] <... futex resumed>) = ?
[pid 1862681] +++ exited with 128 +++
[pid 1862679] +++ exited with 128 +++
[pid 1862680] +++ exited with 128 +++
[pid 1862682] +++ exited with 128 +++
+++ exited with 128 +++
Removing the network = "host" configuration and restarting containerd resolved the error, and kubectl top pods started working.
Steps to reproduce
/etc/containerd/config.toml:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
TypeUrl = "io.containerd.runsc.v1.options"
ConfigPath = "/etc/containerd/runsc.toml"
SystemdCgroup = true/etc/containerd/runsc.toml:
[runsc_config]
network = "host"
systemd-cgroup = "true"
runsc version
runsc version release-20260302.0
spec: 1.1.0-rc.1docker version (if using docker)
containerd github.com/containerd/containerd 1.7.29+unknownuname
Linux node 6.6.110-42.4.tl4.x86_64 #1 SMP Fri Dec 12 16:15:13 CST 2025 x86_64 x86_64 x86_64 GNU/Linux
kubectl (if using Kubernetes)
Client Version: v1.35.0
Kustomize Version: v5.7.1
Server Version: v1.35.0repo state (if built from source)
No response
runsc debug logs (if available)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
type: bugSomething isn't workingSomething isn't working