AppVM fails to start with "device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0"

[fishilico]

Qubes OS version: R4.0-rc5

Affected component(s):

• Default AppVM (personal, work, ...)

---

Steps to reproduce the behavior:

• Install Qubes OS version R4.0-rc5 on a Dell ThinkPad T470 laptop (using instructions from https://www.qubes-os.org/doc/thinkpad-troubleshooting/ in order to boot in UEFI mode).
• Boot Qubes OS and create all the default qubes.
• Log in.
• sys-net, sys-usb and sys-firewall are running (I have access to the Internet in sys-net) but "personal" AppVM fails to start. Qubes Manager gives an error message.
• Disable networking on personal AppVM and start it. It runs fine.
• Shutdown personal and re-enable networking on it. It now fails to start.
• Reboot the whole system. Now everything works fine :)

Expected behavior:

personal AppVM runs with networking, without needed to reboot the computer.

Actual behavior:

personal AppVM runs without networking but fails to start with networking right after booting Qubes for the first time. This fails with both fedora-26 and debian-9 templates, and both PVH and HVM modes.

General notes:

Qube Manager shows the following message box when personal AppVM fails to start:

ERROR: Start failed: internal error: libxenlight failed to create new domain 'testing-debian', see /var/log/libvirt/libxl/libxl-driver.log for details

This log files contains:

2018-03-13 14:59:18.943+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:18.943+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add nic devices
2018-03-13 14:59:29.006+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:29.011+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 18

By the way, /var/log/qubes/qubes.log contains:

2018-03-13 15:59:05,903 b'  WARNING: Sum of all thin volume sizes (639.15 GiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (476.74 GiB)!\n'

I am not sure whether this warning is related with the issue.

---

Related issues:

[fishilico]

After asking for help on IRC, I tried to reboot my computer and the issue disappeared. I reported the issue nonetheless in order to help other people who may encounter it.

I would also be very interested in knowing what commands I could have used in order to debug the issue more and find out what caused it (or what commands will be helpful if the issue happens again).

[fishilico]

After asking for help on IRC, I tried to reboot my computer and the issue disappeared. I reported the issue nonetheless in order to help other people who may encounter it.

I would also be very interested in knowing what commands I could have used in order to debug the issue more and find out what caused it (or what commands will be helpful if the issue happens again).

[marmarek]

One thing to check is if xendriverdomain.service is running in sys-firewall. Also it would be helpful to know if libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0 line appears immediately at VM startup try, or after some timeout (10s? 30s?).

[marmarek]

One thing to check is if xendriverdomain.service is running in sys-firewall. Also it would be helpful to know if libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0 line appears immediately at VM startup try, or after some timeout (10s? 30s?).

[fishilico]

I have posted the full logs of running tail -F on all .log files in /var/log on https://pastebin.com/4DhLMe0W , when I tried starting an AppVM named "testing-debian" created from debian-9 template, in PVH mode. Here are the relevant lines (with times):

==> /var/log/qubes/qubes.log <==
2018-03-13 15:59:05,389 Starting testing-debian

==> /var/log/libvirt/libxl/libxl-driver.log <==
2018-03-13 14:59:18.943+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:18.943+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add nic devices
2018-03-13 14:59:29.006+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:29.011+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 18
 
==> /var/log/qubes/qubes.log <==
2018-03-13 15:59:29,117 Start failed: internal error: libxenlight failed to create new domain 'testing-debian'

Therefore libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0 appeared 13 seconds after starting the VM and there were 10 seconds before libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/7/backend/vif/18/0.

[fishilico]

I have posted the full logs of running tail -F on all .log files in /var/log on https://pastebin.com/4DhLMe0W , when I tried starting an AppVM named "testing-debian" created from debian-9 template, in PVH mode. Here are the relevant lines (with times):

==> /var/log/qubes/qubes.log <==
2018-03-13 15:59:05,389 Starting testing-debian

==> /var/log/libvirt/libxl/libxl-driver.log <==
2018-03-13 14:59:18.943+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:18.943+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add nic devices
2018-03-13 14:59:29.006+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/7/backend/vif/18/0
2018-03-13 14:59:29.011+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 18
 
==> /var/log/qubes/qubes.log <==
2018-03-13 15:59:29,117 Start failed: internal error: libxenlight failed to create new domain 'testing-debian'

Therefore libxl_device.c:1081:device_backend_callback: unable to add device with path /local/domain/7/backend/vif/18/0 appeared 13 seconds after starting the VM and there were 10 seconds before libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/7/backend/vif/18/0.