mknod /dev/net/tun read access fails after update to v1.7.24

[weikinhuang]

Description

I believe there was a change somewhere in containerd or runc where created devices might be missing the DeviceAllow configuration where it was previously working for devices created within the container.

I am running this under kubernetes v1.31.3

Steps to reproduce the issue

1. Create a pod spec with the following container
2. Install openvpn in the container
3. Create tun device
4. mkdir -p /dev/net
5. mknod -m 0666 /dev/net/tun c 10 200
6. Start openvpn with the tun device

  - args:
    - sleep infinity
    env:
    image: debian:latest
    imagePullPolicy: IfNotPresent
    name: ingress-vpn
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        - MKNOD
        - SETGID
        - SETUID
        drop:
        - all

Describe the results you received and expected

+ firewall::iptables-append -A POSTROUTING -t nat -s 192.168.255.0/24 -o tun0 -j MASQUERADE
+ shift
+ iptables -C POSTROUTING -t nat -s 192.168.255.0/24 -o tun0 -j MASQUERADE
+ return 0
+ set +x
+ firewall::iptables-append -A POSTROUTING -t nat -s 192.168.254.0/24 -o tun0 -j MASQUERADE
+ shift
+ iptables -C POSTROUTING -t nat -s 192.168.254.0/24 -o tun0 -j MASQUERADE
+ return 0
+ set +x
2024/11/30 00:42:45 [INFO] generate received request
2024/11/30 00:42:45 [INFO] received CSR
2024/11/30 00:42:45 [INFO] generating key: ecdsa-256
2024/11/30 00:42:45 [INFO] encoded CSR
2024/11/30 00:42:45 [INFO] signed certificate with serial number [REDACTED]
2024-11-30 00:42:45 WARNING: --topology net30 support for server configs with IPv4 pools will be removed in a future release. Please migrate to --topology subnet as soon as possible.
2024-11-30 00:42:45 DEPRECATED OPTION: --cipher set to 'none' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 
2024-11-30 00:42:45 OpenVPN 2.6.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2024-11-30 00:42:45 library versions: OpenSSL 3.0.11 19 Sep 2023, LZO 2.10
2024-11-30 00:42:45 DCO version: N/A
2024-11-30 00:42:45 WARNING: using --duplicate-cn and --client-config-dir together is probably not what you want
2024-11-30 00:42:45 Note: OpenSSL hardware crypto engine functionality is not available
2024-11-30 00:42:45 ERROR: Cannot open TUN/TAP dev /dev/net/tun: Operation not permitted (errno=1)
2024-11-30 00:42:45 Exiting due to fatal error

What version of containerd are you using?

v1.7.24

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[Phil57]

Having the same issue with raw Docker and Gluetun.
Downgrading to 1.7.23 fixes it.

INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
INFO [routing] routing cleanup...
INFO [routing] default route found: interface eth0, gateway 172.20.0.1, assigned IP 172.20.0.2 and family v4
INFO [routing] deleting route for 0.0.0.0/0
ERROR creating tun device: unix opening TUN device file: operation not permitted

[AkihiroSuda]

Downgrading to 1.7.23 fixes it.

Is this an issue of runc binary included in containerd, or an issue of containerd itself?

[Phil57]

Downgrading to 1.7.23 fixes it.

Is this an issue of runc binary included in containerd, or an issue of containerd itself?

How can I know / test that ?

[AkihiroSuda]

Downgrading to 1.7.23 fixes it.

Is this an issue of runc binary included in containerd, or an issue of containerd itself?

How can I know / test that ?

• Does this issue happen with containerd v1.7.23 with runc v1.2.x? If yes, this is a regression in runc, not in containerd
• What about contained v1.7.24 with runc v1.1.x? If yes, this is a regression in containerd

[Phil57]

╰─❯ runc -v runc version 1.1.14 commit: v1.1.14-0-g2c9f560 spec: 1.0.2-dev go: go1.22.9 libseccomp: 2.5.5

On both machines, OS is Ubuntu 24.04 and runc 1.1.14.
The only package uptated at the time of the issue is containerd.

[jumoog]

It happend here: opencontainers/runc@2ce40b6

Obviously, a container should not have access to tun/tap device, unless
it is explicitly specified in configuration.
Now, removing this might create a compatibility issue, but I see no
other choice.

[weikinhuang]

That commit happened in the change to v1.2.0, but this started failing between v1.2.1 and v1.2.2. I'm guessing the solution is to pass the /dev/net/tun device to the kubernetes pod?

My solution was to use smarter-device-manager to setup /dev/net/tun as a resource request. Mounting it as a char device volume did not work.

[jumoog]

No, from 1.7.23 to 1.7.24 containerd release, runc has been upgraded from v1.1.14 to v1.2.1 710cd37#diff-8825467b414b34c7c6b0ec0656aa20a569df93e70e6d55cfa5a7721d78ce36d9L1

[weikinhuang]

Got it, I was just checking the release notes, and the cherry pick's diff has it from 1.2.1 to 1.2.2.