How to enable privileged for k8s + containerd + kata to access host devices ?

[LiangZhou-CTY]

Hi there,

I followed the guide https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/privileged.md#host-devices to enable privileged feature on kata but failed.

Here's my steps:

1. set privileged to true on my pod yaml:

[root@localhost kata]# cat test-kata.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-kata
spec:
  runtimeClassName: kata
  containers:
  - name: test-kata-1
    securityContext:
      privileged: true
    image: ubuntu

2. set related fields in /etc/containerd/config.toml :

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
            runtime_type = "io.containerd.kata.v2"
            privileged_without_host_devices = true

3. run the pod, but cannot get the host devices under /dev:

[root@localhost kata]# kubectl exec test-kata -ti -- /bin/bash
root@test-kata:/#
root@test-kata:/# uname -a
Linux test-kata 5.4.71 #1 SMP Fri Apr 9 19:03:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@test-kata:/# ls /dev
fd  full  mqueue  null  ptmx  pts  random  shm  stderr  stdin  stdout  termination-log  tty  urandom  zero
root@test-kata:/#

we see that the host devices cannot pass through to kata container.

4. I tried the same steps on runc, it works:

[root@localhost runc]# kubectl exec -ti liang -- /bin/sh
# uname -a
Linux liang 4.18.0-240.el8.x86_64 #1 SMP Fri Sep 25 19:48:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# ls /dev
autofs           fb0      kmsg          mcelog  ppp     sda1      snd              tty0   tty16  tty23  tty30  tty38  tty45  tty52  tty6   ttyS0    usbmon1  vcs6   vfio
bsg              fd       kvm           mem     ptmx    sda2      sr0              tty1   tty17  tty24  tty31  tty39  tty46  tty53  tty60  ttyS1    usbmon2  vcsa   vga_arbiter
bus              full     loop-control  midi    pts     sda3      sr1              tty10  tty18  tty25  tty32  tty4   tty47  tty54  tty61  ttyS2    vcs      vcsa1  vhci
core             fuse     lp0           mqueue  random  sg0       stderr           tty11  tty19  tty26  tty33  tty40  tty48  tty55  tty62  ttyS3    vcs1     vcsa2  vhost-net
cpu              hidraw0  lp1           net     raw     sg1       stdin            tty12  tty2   tty27  tty34  tty41  tty49  tty56  tty63  uhid     vcs2     vcsa3  vhost-vsock
cpu_dma_latency  hpet     lp2           null    rfkill  sg2       stdout           tty13  tty20  tty28  tty35  tty42  tty5   tty57  tty7   uinput   vcs3     vcsa4  vmci
dmmidi           hwrng    lp3           nvram   rtc0    shm       termination-log  tty14  tty21  tty29  tty36  tty43  tty50  tty58  tty8   urandom  vcs4     vcsa5  vsock
dri              input    mapper        port    sda     snapshot  tty              tty15  tty22  tty3   tty37  tty44  tty51  tty59  tty9   usbmon0  vcs5     vcsa6  zero
#

5. I searched this issue, seems there's a similar discussion but that discussion is not the same as my one. That one is about setting "privileged_without_host_devices = false" and got container startup failure.
   privileged securityContext is causing pod to fail on k8s #1342

Anything I missed ?

Thanks.

[egernst]

You’ll want to explicitly add those devices if you need them. Containerd won’t add these devices if you have set: privileged_without_host_devices = true

[LiangZhou-CTY]

You’ll want to explicitly add those devices if you need them. Containerd won’t add these devices if you have set: privileged_without_host_devices = true

Thanks @egernst for your quick response.
I'm not very familar with k8s, is there any example for kata to "explicitily add those devices" ?

[c3d]

@egernst At present, the documentation does not give examples (not even a suggestion) that the /dev/ entries need to be manually passed in that case.

We provide a number of configuration files, but we should also provide a k8s yaml file showing what to do in that case, no?