Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubevirt virt-handler Init:CrashLoopBackOff #7771

Closed
Zigko opened this issue May 18, 2022 · 15 comments
Closed

Kubevirt virt-handler Init:CrashLoopBackOff #7771

Zigko opened this issue May 18, 2022 · 15 comments
Labels
kind/bug lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@Zigko
Copy link

Zigko commented May 18, 2022

What happened:
I was installing kubevirt operator following the tutorial in this page:
https://kubevirt.io/2019/KubeVirt_k8s_crio_from_scratch_installing_KubeVirt.html

But the pod handler wont start and I don't understand the reason...

root@debian:~# **kubectl get pods -n kubevirt**
NAME                               READY   STATUS       RESTARTS         AGE
virt-api-77df5c4f87-7mqv4          1/1     Running      1 (17m ago)      27m
virt-api-77df5c4f87-wcq44          1/1     Running      1 (17m ago)      27m
virt-controller-749d8d99d4-56gb7   1/1     Running      1 (17m ago)      27m
virt-controller-749d8d99d4-78j6x   1/1     Running      1 (17m ago)      27m
virt-handler-4w99d                 0/1     Init:Error   14 (5m18s ago)   27m
virt-operator-564f568975-g9wh4     1/1     Running      1 (17m ago)      31m
virt-operator-564f568975-wnpz8     1/1     Running      1 (17m ago)      31m

root@debian:~# **kubectl logs virt-handler-4w99d -n kubevirt**
Error from server (BadRequest): container "virt-handler" in pod "virt-handler-4w99d" is waiting to start: PodInitializing

kubectl describe pod virt-handler-4w99d -n kubevirt

root@debian:~# kubectl describe pod virt-handler-4w99d -n kubevirt
Name:                 virt-handler-4w99d
Namespace:            kubevirt
Priority:             1000000000
Priority Class Name:  kubevirt-cluster-critical
Node:                 debian/172.16.16.13
Start Time:           Wed, 18 May 2022 16:33:05 +0100
Labels:               app.kubernetes.io/component=kubevirt
                      app.kubernetes.io/managed-by=virt-operator
                      app.kubernetes.io/version=v0.52.0
                      controller-revision-hash=f68858c57
                      kubevirt.io=virt-handler
                      pod-template-generation=1
                      prometheus.kubevirt.io=true
Annotations:          cni.projectcalico.org/containerID: 97dd02deebc33d2714172adde8aaae83a3d33d668d3555057012b74f3717e25f
                      cni.projectcalico.org/podIP: 192.168.245.224/32
                      cni.projectcalico.org/podIPs: 192.168.245.224/32
                      kubevirt.io/install-strategy-identifier: 72d62fe25180ebc296d7a30b4ba2508933d9c2fe
                      kubevirt.io/install-strategy-registry: quay.io/kubevirt
                      kubevirt.io/install-strategy-version: v0.52.0
Status:               Pending
IP:                   192.168.245.224
IPs:
  IP:           192.168.245.224
Controlled By:  DaemonSet/virt-handler
Init Containers:
  virt-launcher:
    Container ID:  containerd://8a3b93bab9cafb06ae1e4cd0ab7cae040e87cf88b0cb7af92b5029bac23c8e0e
    Image:         quay.io/kubevirt/virt-launcher:v0.52.0
    Image ID:      quay.io/kubevirt/virt-launcher@sha256:7138d7de949a86955718e07edb90381b3abf1dd2e642d55c0db66fb15b21719b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      node-labeller.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 18 May 2022 17:00:28 +0100
      Finished:     Wed, 18 May 2022 17:00:28 +0100
    Ready:          False
    Restart Count:  14
    Environment:    <none>
    Mounts:
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5spst (ro)
Containers:
  virt-handler:
    Container ID:
    Image:         quay.io/kubevirt/virt-handler:v0.52.0
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      virt-handler
      --port
      8443
      --hostname-override
      $(NODE_NAME)
      --pod-ip-address
      $(MY_POD_IP)
      --max-metric-requests
      3
      --console-server-port
      8186
      --graceful-shutdown-seconds
      315
      -v
      2
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      10m
      memory:   230Mi
    Liveness:   http-get https://:8443/healthz delay=15s timeout=10s period=45s #success=1 #failure=3
    Readiness:  http-get https://:8443/healthz delay=15s timeout=10s period=20s #success=1 #failure=3
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      MY_POD_IP:   (v1:status.podIP)
    Mounts:
      /etc/podinfo from podinfo (rw)
      /etc/virt-handler/clientcertificates from kubevirt-virt-handler-certs (ro)
      /etc/virt-handler/servercertificates from kubevirt-virt-handler-server-certs (ro)
      /pods from kubelet-pods-shortened (rw)
      /profile-data from profile-data (rw)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/lib/kubelet/pods from kubelet-pods (rw)
      /var/lib/kubevirt from virt-lib-dir (rw)
      /var/lib/kubevirt-node-labeller from node-labeller (rw)
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-libvirt-runtimes from libvirt-runtimes (rw)
      /var/run/kubevirt-private from virt-private-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5spst (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kubevirt-virt-handler-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-certs
    Optional:    true
  kubevirt-virt-handler-server-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-virt-handler-server-certs
    Optional:    true
  profile-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  libvirt-runtimes:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-libvirt-runtimes
    HostPathType:
  virt-share-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt
    HostPathType:
  virt-lib-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubevirt
    HostPathType:
  virt-private-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt-private
    HostPathType:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:
  kubelet-pods-shortened:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
  kubelet-pods:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:
  node-labeller:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubevirt-node-labeller
    HostPathType:
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['k8s.v1.cni.cncf.io/network-status'] -> network-status
  kube-api-access-5spst:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                  From               Message
  ----     ------          ----                 ----               -------
  Normal   Scheduled       32m                  default-scheduler  Successfully assigned kubevirt/virt-handler-4w99d to debian
  Normal   Pulled          30m (x5 over 32m)    kubelet            Container image "quay.io/kubevirt/virt-launcher:v0.52.0" already present on machine
  Normal   Created         30m (x5 over 32m)    kubelet            Created container virt-launcher
  Normal   Started         30m (x5 over 32m)    kubelet            Started container virt-launcher
  Warning  BackOff         27m (x25 over 32m)   kubelet            Back-off restarting failed container
  Normal   SandboxChanged  21m (x2 over 22m)    kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          19m (x4 over 21m)    kubelet            Container image "quay.io/kubevirt/virt-launcher:v0.52.0" already present on machine
  Normal   Created         19m (x4 over 21m)    kubelet            Created container virt-launcher
  Normal   Started         19m (x4 over 21m)    kubelet            Started container virt-launcher
  Warning  BackOff         102s (x93 over 21m)  kubelet            Back-off restarting failed container

What you expected to happen:
Resolve the problem

Environment:

  • KubeVirt version (use virtctl version): v0.52.0
  • Kubernetes version (use kubectl version): v1.23.6
  • VM or VMI specifications: N/A
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 11 (bullseye)
  • Kernel (e.g. uname -a): 5.10.0-13-amd64
  • Install tools: N/A
  • Others: N/A
@Zigko Zigko added the kind/bug label May 18, 2022
@vasiliy-ul
Copy link
Contributor

It looks like the init container fails. Could you provide the logs?

kubectl logs -n kubevirt virt-handler-4w99d -c virt-launcher

Do you have apparmor running on your host?

@Zigko
Copy link
Author

Zigko commented May 18, 2022 

root@debian:~# kubectl logs -n kubevirt virt-handler-4w99d -c virt-launcher
error: failed to get emulator capabilities
error: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing: libvirt:  error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied

No i dont have that, do i need it?

@vasiliy-ul
Copy link
Contributor

Permission denied

I actually think you do have it. Check you system logs. Not sure where exactly you need to look in Debian. Perhaps grep DEN /var/log/audit/audit.log or journalctl -b | grep DEN. Also you can run aa-status.

@vasiliy-ul
Copy link
Contributor

If its apparmor you can try to work around this issue with aa-disable /usr/sbin/libvirtd. But this is not a good solution for "production" as it will disable the security profile for libvirtd. Though should be fine for testing purposes.

@Zigko
Copy link
Author

Zigko commented May 18, 2022 •
edited
Loading

This is what i get:

root@debian:~# journalctl -b | grep DEN
May 18 16:42:55 debian audit[655]: AVC apparmor="DENIED" operation="capable" profile="/snap/core/12834/usr/lib/snapd/snap-confine" pid=655 comm="snap-confine" capability=12  capname="net_admin"
May 18 16:42:55 debian audit[655]: AVC apparmor="DENIED" operation="capable" profile="/snap/core/12834/usr/lib/snapd/snap-confine" pid=655 comm="snap-confine" capability=38  capname="perfmon"
May 18 16:42:57 debian audit[655]: AVC apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/etc/ssh/ssh_config" pid=655 comm=5468726561642028706F6F6C656429 requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 18 16:42:57 debian kernel: audit: type=1400 audit(1652888577.188:37): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/etc/ssh/ssh_config" pid=655 comm=5468726561642028706F6F6C656429 requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 18 16:42:58 debian audit[655]: AVC apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/etc/ssh/ssh_config" pid=655 comm=5468726561642028706F6F6C656429 requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 18 16:42:58 debian kernel: audit: type=1400 audit(1652888578.775:38): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/etc/ssh/ssh_config" pid=655 comm=5468726561642028706F6F6C656429 requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 18 16:44:20 debian audit[6316]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=6316 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:44:20 debian kernel: audit: type=1400 audit(1652888660.539:39): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=6316 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:44:36 debian audit[6825]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=6825 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:44:36 debian kernel: audit: type=1400 audit(1652888676.138:40): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=6825 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:45:00 debian audit[7227]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=7227 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:45:00 debian kernel: audit: type=1400 audit(1652888700.122:41): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=7227 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:45:49 debian audit[7926]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=7926 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:45:49 debian kernel: audit: type=1400 audit(1652888749.162:42): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=7926 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:47:15 debian audit[9081]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=9081 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:47:15 debian kernel: audit: type=1400 audit(1652888835.184:43): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=9081 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:50:07 debian audit[11328]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=11328 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:50:07 debian kernel: audit: type=1400 audit(1652889007.142:44): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=11328 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:55:13 debian audit[15039]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=15039 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 16:55:13 debian kernel: audit: type=1400 audit(1652889313.154:45): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=15039 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:00:28 debian audit[18882]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=18882 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:00:28 debian kernel: audit: type=1400 audit(1652889628.159:46): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=18882 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:05:39 debian audit[22716]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=22716 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:05:39 debian kernel: audit: type=1400 audit(1652889939.146:47): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=22716 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:10:46 debian audit[26489]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=26489 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:10:46 debian kernel: audit: type=1400 audit(1652890246.143:48): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=26489 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:15:58 debian audit[30271]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=30271 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:15:58 debian kernel: audit: type=1400 audit(1652890558.151:49): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=30271 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:21:02 debian audit[34013]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=34013 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:21:02 debian kernel: audit: type=1400 audit(1652890862.175:50): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=34013 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:26:03 debian audit[38640]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=38640 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:26:03 debian kernel: audit: type=1400 audit(1652891163.159:51): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=38640 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:31:11 debian audit[42392]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=42392 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:31:11 debian kernel: audit: type=1400 audit(1652891471.167:52): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=42392 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:36:15 debian audit[46027]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=46027 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:36:15 debian kernel: audit: type=1400 audit(1652891775.164:53): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=46027 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:41:26 debian audit[49837]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=49837 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:41:26 debian kernel: audit: type=1400 audit(1652892086.161:54): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=49837 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:46:37 debian audit[53641]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=53641 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:46:37 debian kernel: audit: type=1400 audit(1652892397.157:55): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=53641 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:51:40 debian audit[57295]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=57295 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:51:40 debian kernel: audit: type=1400 audit(1652892700.137:56): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=57295 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:56:44 debian audit[60947]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=60947 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 17:56:44 debian kernel: audit: type=1400 audit(1652893004.158:57): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=60947 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:01:49 debian audit[64668]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=64668 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:01:49 debian kernel: audit: type=1400 audit(1652893309.149:58): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=64668 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:06:51 debian audit[68290]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=68290 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:06:51 debian kernel: audit: type=1400 audit(1652893611.164:59): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=68290 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:11:53 debian audit[71923]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=71923 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:11:53 debian kernel: audit: type=1400 audit(1652893913.160:60): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=71923 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:16:58 debian audit[75643]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=75643 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:16:58 debian kernel: audit: type=1400 audit(1652894218.171:61): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=75643 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:22:08 debian audit[79397]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=79397 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:22:08 debian kernel: audit: type=1400 audit(1652894528.167:62): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=79397 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:27:15 debian audit[83060]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=83060 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:27:15 debian kernel: audit: type=1400 audit(1652894835.159:63): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=83060 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:32:21 debian audit[86787]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=86787 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:32:21 debian kernel: audit: type=1400 audit(1652895141.163:64): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=86787 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:37:24 debian audit[90453]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=90453 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:37:24 debian kernel: audit: type=1400 audit(1652895444.151:65): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=90453 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:42:33 debian audit[94200]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=94200 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:42:33 debian kernel: audit: type=1400 audit(1652895753.155:66): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=94200 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:47:37 debian audit[97965]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=97965 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:47:37 debian kernel: audit: type=1400 audit(1652896057.167:67): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=97965 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:52:43 debian audit[101623]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=101623 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:52:43 debian kernel: audit: type=1400 audit(1652896363.151:68): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=101623 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:57:44 debian audit[105267]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=105267 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 18:57:44 debian kernel: audit: type=1400 audit(1652896664.155:69): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=105267 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:02:49 debian audit[108965]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=108965 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:02:49 debian kernel: audit: type=1400 audit(1652896969.147:70): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=108965 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:07:55 debian audit[112619]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=112619 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:07:55 debian kernel: audit: type=1400 audit(1652897275.174:71): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=112619 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:12:57 debian audit[116298]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=116298 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:12:57 debian kernel: audit: type=1400 audit(1652897577.134:72): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=116298 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:18:00 debian audit[119974]: AVC apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=119974 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0
May 18 19:18:00 debian kernel: audit: type=1400 audit(1652897880.162:73): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/qemu-kvm" pid=119974 comm="rpc-worker" requested_mask="x" denied_mask="x" fsuid=107 ouid=0

Beside the aa-disable /usr/sbin/libvirtd what could I do to be able to use kubevirt?
Because I cant run aa-disable:

root@debian:~# aa-
aa-enabled         aa-exec            aa-remove-unknown  aa-status          aa-teardown

@ikandars
Copy link

You can add /usr/libexec/qemu-kvm PUx into /etc/apparmor.d/usr.sbin.libvirtd file then reload it

sudo systemctl reload apparmor.service

It fix my issue in Ubuntu 20.04 and found the solution from this thread #4303

@vasiliy-ul
Copy link
Contributor

Because I cant run aa-disable:

You need to install apparmor-utils package (not 100% sure if its called like that in Debian though)

BUT...

You can add /usr/libexec/qemu-kvm PUx into /etc/apparmor.d/usr.sbin.libvirtd file then reload it

sudo systemctl reload apparmor.service

It fix my issue in Ubuntu 20.04 and found the solution from this thread #4303

Yep, that would probably be a better solution 👍

@Zigko
Copy link
Author

Zigko commented May 19, 2022

Thanks, now its working

root@debian:~# kubectl get pods -n kubevirt
NAME                               READY   STATUS    RESTARTS      AGE
virt-api-77df5c4f87-7mqv4          1/1     Running   2 (87m ago)   19h
virt-api-77df5c4f87-wcq44          1/1     Running   2 (87m ago)   19h
virt-controller-749d8d99d4-56gb7   1/1     Running   2 (87m ago)   19h
virt-controller-749d8d99d4-78j6x   1/1     Running   2 (87m ago)   19h
virt-handler-4w99d                 1/1     Running   1 (87m ago)   19h
virt-operator-564f568975-g9wh4     1/1     Running   2 (87m ago)   19h
virt-operator-564f568975-wnpz8     1/1     Running   2 (87m ago)   19h

@h1r0mu
Copy link

h1r0mu commented Jun 19, 2022

Hi,
I faced the exact same problem, but the solution in #7771 (comment) didn't work for my environment.

I finally got it to work by using aa-logprof command and selecting Inherit to generate an apparmor profile though I'm not sure whether it's a correct solution or not.

$ sudo aa-logprof
Reading log entries from /var/log/syslog.
Updating AppArmor profiles in /etc/apparmor.d.

Profile:  /usr/sbin/libvirtd
Execute:  /usr/libexec/qemu-kvm
Severity: unknown

(I)nherit / (C)hild / (P)rofile / (N)amed / (U)nconfined / (X) ix On / (D)eny / Abo(r)t / (F)inish
Complain-mode changes:
Enforce-mode changes:

@kubevirt-bot
Copy link
Contributor

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@kubevirt-bot kubevirt-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 17, 2022
@xzycn
Copy link

xzycn commented Sep 20, 2022

This issue will be fixed in future release, right?

@vasiliy-ul
Copy link
Contributor

I am not aware if someone is working on that ATM. But for sure, Apparmor support in KubeVirt will definitely bring a lot of value to the project. Contributions are more than welcome here.

@kubevirt-bot
Copy link
Contributor

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

@kubevirt-bot kubevirt-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 20, 2022
@vasiliy-ul vasiliy-ul mentioned this issue Oct 28, 2022
Closed
@andreabolognani andreabolognani mentioned this issue Nov 9, 2022
Open
@kubevirt-bot
Copy link
Contributor

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

@kubevirt-bot
Copy link
Contributor

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

node-labeller.sh: Consider AppArmor restrictions vasiliy-ul/kubevirt
6 participants
@ikandars @xzycn @h1r0mu @kubevirt-bot @Zigko @vasiliy-ul