sys-usb does not start anymore - libxl: no permission to assign

[schnurentwickler]

Qubes OS version:

R4.0 rc4

Affected TemplateVMs:

sys-usb, newly usb-based AppVMs, (dom0?)

Steps to reproduce the behavior:

Use the Qubes-Manager -> Devices and unselect one usb controller. (until this task, qubes worked well after installation for some days)
After the error came up, I used the console command: qvm-pci a -p sys-usb dom0:00_1a.0 (as well with 00_1d.0).
Even later: Undo all changes back to qubes standard. Also use all usb devices and card-reader. (attached with console commands)

Expected behavior:

usb AppVMs starts as they should right after configuration.

Actual behavior:

Cannot start any usb controller attached HVM.

General notes:

Without any usb controller attached, sys-usb starts via console command, but outputs it cannot execute qrexec-daemon, which results in a not-running VM.
Same behavior with newly created HVMs. I can attach/detach usb devices on new created HVMs which results in a (not) working qube.

I can create and start network device attached VMs, I can start newly created usb HVMs without any usb controller but card-reader attached.

Full error message: Start failed: internal error: libxenlight failed to create new domain 'usb2Go', see /var/log/libvirt/libxl/libxl-driver.log for details.

/var/log/libvirt/libxl/libxl-driver.log:

2018-02-19 12:03:23.216+0000: libxl: libxl_pci.c:1134:do_pci_add: xc_assign_device failed: Operation not permitted
2018-02-19 12:03:23.216+0000: libxl: libxl_pci.c:1338:libxl__add_pcidevs: libxl_device_pci_add failed: -3
2018-02-19 12:03:23.216+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add pci devices
2018-02-19 12:28:17.802+0000: libxl: libxl_pci.c:1134:do_pci_add: xc_assign_device failed: Operation not permitted
2018-02-19 12:28:17.802+0000: libxl: libxl_pci.c:1338:libxl__add_pcidevs: libxl_device_pci_add failed: -3
2018-02-19 12:28:17.802+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add pci devices
2018-02-19 12:41:20.516+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/5/backend/vif/20/0
2018-02-19 12:41:20.526+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 20
2018-02-19 12:41:20.726+0000: libxl: libxl_device.c:1081:device_backend_callback: unable to remove device with path /local/domain/5/backend/vif/19/0
2018-02-19 12:41:20.736+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 19
2018-02-19 12:41:56.657+0000: libxl: libxl_pci.c:1134:do_pci_add: xc_assign_device failed: Operation not permitted
2018-02-19 12:41:56.657+0000: libxl: libxl_pci.c:1338:libxl__add_pcidevs: libxl_device_pci_add failed: -3
2018-02-19 12:41:56.657+0000: libxl: libxl_create.c:1512:domcreate_attach_devices: unable to add pci devices

ps: why is 'copy to qubes clipboard'-shortcut not working for an dom0 focused window. (only copy) :-(

Related issues:

Maybe #1544

[schnurentwickler]

The folder structure /local/domain/5/backend/vif/... in the error logfile (shown above) looks quite suspicious. Shall it start with /local/...?

More debug: I already restarted and also enabled autostart on sys-usb. (was disabled but worked until I used qubes-manager)
I got sys-usb to start without any usb-controller but card reader attached (should be an usb device) and it is accessible without any qrexec errors.

[schnurentwickler]

The folder structure /local/domain/5/backend/vif/... in the error logfile (shown above) looks quite suspicious. Shall it start with /local/...?

More debug: I already restarted and also enabled autostart on sys-usb. (was disabled but worked until I used qubes-manager)
I got sys-usb to start without any usb-controller but card reader attached (should be an usb device) and it is accessible without any qrexec errors.

[marmarek]

See /var/log/xen/console/hypervisor.log, to confirm, but it indeed looks like the same as #1544
There is an option to bypass that check: add -o no-strict-reset=True when attaching device.

[marmarek]

See /var/log/xen/console/hypervisor.log, to confirm, but it indeed looks like the same as #1544
There is an option to bypass that check: add -o no-strict-reset=True when attaching device.

[schnurentwickler]

(better not to display the complete hypervisor.log)

(XEN) Scrubbing Free RAM on 1 nodes using 2 CPUs
(XEN) ...................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 312kB init memory
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom6.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom6 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom8.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom8 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom10.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom10 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom17.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom17 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom23.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom23 failed (-1)

But I do not get it. It worked well after installation without any modification. :-/
Does this mean the qubes installation process always attaches usb devices with -o no-strict-reset=True ?

[schnurentwickler]

(better not to display the complete hypervisor.log)

(XEN) Scrubbing Free RAM on 1 nodes using 2 CPUs
(XEN) ...................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 312kB init memory
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom6.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom6 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom8.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom8 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom10.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom10 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom17.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom17 failed (-1)
(XEN) [VT-D] It's disallowed to assign 0000:00:1d.0 with shared RMRR at ca9c6000 for Dom23.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1d.0 to dom23 failed (-1)

But I do not get it. It worked well after installation without any modification. :-/
Does this mean the qubes installation process always attaches usb devices with -o no-strict-reset=True ?

[schnurentwickler]

I can confirm that it is working. sys-usb is starting with even one attached usb controller.
This is maybe a work around, because then my wifi cards AND card reader are all pci devices or at least the card reader uses another usb controller chip.
All of them are attached separately to newly created HVMs and/or got attached/detached to other HVMs.

For now it works and the usb VMs are not that important to have to work with strict-reset.
Unfortunately I could not find this option anywhere and strict-reset options I have found only for qvm-prefs in qubes documentation. Even qvm-pci --help does not show these special options. (maybe with notes to use strict-reset with care and only if necessary)
Never mind. I have found this information in the man page of qvm-pci, but without the hint to activate the pci option with -o. So I would have to guess how to use it, because without -o argument an error message appears.

Can be closed if no other cases occur.

[schnurentwickler]

I can confirm that it is working. sys-usb is starting with even one attached usb controller.
This is maybe a work around, because then my wifi cards AND card reader are all pci devices or at least the card reader uses another usb controller chip.
All of them are attached separately to newly created HVMs and/or got attached/detached to other HVMs.

For now it works and the usb VMs are not that important to have to work with strict-reset.
Unfortunately I could not find this option anywhere and strict-reset options I have found only for qvm-prefs in qubes documentation. Even qvm-pci --help does not show these special options. (maybe with notes to use strict-reset with care and only if necessary)
Never mind. I have found this information in the man page of qvm-pci, but without the hint to activate the pci option with -o. So I would have to guess how to use it, because without -o argument an error message appears.

Can be closed if no other cases occur.

[marmarek]

Default sys-usb is configured with no-strict-reset option enabled.

[marmarek]

Default sys-usb is configured with no-strict-reset option enabled.

[andrewdavidwong]

Can be closed if no other cases occur.

Closing this as "resolved." If you believe the issue is not yet resolved, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen this. Thank you.

[andrewdavidwong]

Can be closed if no other cases occur.

Closing this as "resolved." If you believe the issue is not yet resolved, or if anyone is still affected by this issue, please leave a comment, and we'll be happy to reopen this. Thank you.