fail to assign pci device at start of usbvm

[Rudd-O]

I'm experiencing a weird error starting a usbvm:

[user@dom0 ~]$ qvm-start usbvm
--> Creating volatile image: /var/lib/qubes/appvms/usbvm/volatile.img...
--> Loading the VM (type = AppVM)...
Traceback (most recent call last):
  File "/usr/bin/qvm-start", line 125, in <module>
    main()
  File "/usr/bin/qvm-start", line 109, in main
    xid = vm.start(verbose=options.verbose,
preparing_dvm=options.preparing_dvm, start_guid=not options.noguid,
notify_function=tray_notify_generic if options.tray else None)
  File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py",
line 1849, in start
    nd.dettach()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 5249, in
dettach
    if ret == -1: raise libvirtError ('virNodeDeviceDettach() failed')
libvirt.libvirtError: Requested operation is not valid: PCI device
0000:00:1a.0 is in use by driver xenlight, domain usbvm

Restarting libvirtd only aggravates the issue:

[user@dom0 ~]$ qvm-start usbvm
--> Creating volatile image: /var/lib/qubes/appvms/usbvm/volatile.img...
--> Loading the VM (type = AppVM)...
Traceback (most recent call last):
  File "/usr/bin/qvm-start", line 125, in <module>
    main()
  File "/usr/bin/qvm-start", line 109, in main
    xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
  File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1857, in start
    self.libvirt_domain.createWithFlags(libvirt.VIR_DOMAIN_START_PAUSED)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
    if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
libvirt.libvirtError: internal error: libxenlight failed to create new domain 'usbvm'

Weird errors in libxl log:

2015-12-26 x:31:47 TZ libxl: error: libxl_pci.c:1000:do_pci_add: xc_assign_device failed: Operation not permitted
2015-12-26 x:31:47 TZ libxl: error: libxl_create.c:1422:domcreate_attach_pci: libxl_device_pci_add failed: -3

The hypervisor log says:

(XEN) [VT-D] It's disallowed to assign 0000:00:1a.0 with shared RMRR at dbe9a000 for Dom19.
(XEN) XEN_DOMCTL_assign_device: assign 0000:00:1a.0 to dom19 failed (-1)

[Rudd-O]

What's that about the RMRR?

[Rudd-O]

What's that about the RMRR?

[Rudd-O]

Appears to be some new shit:

http://www.gossamer-threads.com/lists/xen/devel/391684

[Rudd-O]

Appears to be some new shit:

http://www.gossamer-threads.com/lists/xen/devel/391684

[marmarek]

The VM has never been started.

Not even using autostart at boot?

What's that about the RMRR?
Appears to be some new shit:
http://www.gossamer-threads.com/lists/xen/devel/391684

We have a way to set rdm_policy=relaxed, bundled with pci_strictreset=false - it should be set by default salt formula for sys-usb, exactly for this reason.
My understanding is that those devices in fact shares some resources, so can't be safely isolated from each other. And Xen doesn't support group assignment (at least for now), so don't know that you are going to assign all such devices to the same VM (which should be safe).

[marmarek]

The VM has never been started.

Not even using autostart at boot?

What's that about the RMRR?
Appears to be some new shit:
http://www.gossamer-threads.com/lists/xen/devel/391684

We have a way to set rdm_policy=relaxed, bundled with pci_strictreset=false - it should be set by default salt formula for sys-usb, exactly for this reason.
My understanding is that those devices in fact shares some resources, so can't be safely isolated from each other. And Xen doesn't support group assignment (at least for now), so don't know that you are going to assign all such devices to the same VM (which should be safe).

[Rudd-O]

Yes, the VM had autostart at boot and the systemd service had failed for this reason.

How do I determine which devices share the RMRR? I couldn't find anything in my logs.

[Rudd-O]

Yes, the VM had autostart at boot and the systemd service had failed for this reason.

How do I determine which devices share the RMRR? I couldn't find anything in my logs.

[Rudd-O]

Holy shit, setting pci_strictreset to False actually let me start that stupid VM!

[Rudd-O]

Holy shit, setting pci_strictreset to False actually let me start that stupid VM!

[marmarek]

I don't know, but guess it is the other USB controller (or USB2.0/USB3.0). If you assign both/all of them to the same VM, you'll see the same address in xen log (assuming you set pci_strictreset=False first, otherwise VM start will fail at the first device...). Yes, kinda ugly way to determine that...

[marmarek]

I don't know, but guess it is the other USB controller (or USB2.0/USB3.0). If you assign both/all of them to the same VM, you'll see the same address in xen log (assuming you set pci_strictreset=False first, otherwise VM start will fail at the first device...). Yes, kinda ugly way to determine that...

[Rudd-O]

Wait, spoke too soon. The VM never ran qrexec-daemon.

[Rudd-O]

Wait, spoke too soon. The VM never ran qrexec-daemon.

[Rudd-O]

It now says in the hypervisor log "It's risky to assign blah blah".

[Rudd-O]

It now says in the hypervisor log "It's risky to assign blah blah".

[Rudd-O]

libxl log:

<date> libxl: error: libxl_device.c:1215:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/24/0 not ready

[Rudd-O]

libxl log:

<date> libxl: error: libxl_device.c:1215:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/24/0 not ready

[marmarek]

Did it crashed just after startup (state "c" on xl list)? If so, probably not enough continuous memory available (take a look #1038 for details) . You can try to free some with xl mem-set 0 <some-number-in-MB> to reduce dom0 memory drastically. For example down to 1500. Sometimes it helps. Otherwise, reboot...

[marmarek]

Did it crashed just after startup (state "c" on xl list)? If so, probably not enough continuous memory available (take a look #1038 for details) . You can try to free some with xl mem-set 0 <some-number-in-MB> to reduce dom0 memory drastically. For example down to 1500. Sometimes it helps. Otherwise, reboot...

[Rudd-O]

Yes. It's state crashed. I just checked.

Assigning all USB devices to the same VM worked to fix the problem.

This sucks. Now I don't have my mouse.

[Rudd-O]

Yes. It's state crashed. I just checked.

Assigning all USB devices to the same VM worked to fix the problem.

This sucks. Now I don't have my mouse.

[Rudd-O]

Note that assigning all USB PCI devices did NOT help start the VM. Even with pci_strictreset set to false. It just killed my mouse.

[Rudd-O]

Note that assigning all USB PCI devices did NOT help start the VM. Even with pci_strictreset set to false. It just killed my mouse.

[Rudd-O]

I will try rebooting now. BRB.

[Rudd-O]

I will try rebooting now. BRB.

[marmarek]

The VM crash at startup is generally a problem with starting VM with PCI devices after some system uptime, memory is much fragmented then. It is independent of previous problem (which is solved with pci_strictreset).

[marmarek]

The VM crash at startup is generally a problem with starting VM with PCI devices after some system uptime, memory is much fragmented then. It is independent of previous problem (which is solved with pci_strictreset).

[Rudd-O]

Alright, excellent. My USB VM has now received the assignment of the two USB PCI devices I intended to isolate (the Bluetooth and camera devices). I still keep the ability to use my mouse. This is GREAT.

Thanks for the pci_strictreset trick.

Improvement: it really should be somehow autodetected whether it is necessary or not.

[Rudd-O]

Alright, excellent. My USB VM has now received the assignment of the two USB PCI devices I intended to isolate (the Bluetooth and camera devices). I still keep the ability to use my mouse. This is GREAT.

Thanks for the pci_strictreset trick.

Improvement: it really should be somehow autodetected whether it is necessary or not.

[marmarek]

It is set for USB VM by salt formula by default.

[marmarek]

It is set for USB VM by salt formula by default.

[Rudd-O]

Yes, that's true, but it would not be the default for a manually created USB VM, which was my case, and I bet the case in many cases. A smart default lower in the stack would reduce the support load.

[Rudd-O]

Yes, that's true, but it would not be the default for a manually created USB VM, which was my case, and I bet the case in many cases. A smart default lower in the stack would reduce the support load.

[marmarek]

The proper solution would be to have PCI group assignment supported by Xen. This way it would detect whether it is really risky to assign particular devices to the VM.

[marmarek]

The proper solution would be to have PCI group assignment supported by Xen. This way it would detect whether it is really risky to assign particular devices to the VM.

[Rudd-O]

Agreed.

[Rudd-O]

Agreed.

[marmarek]

Summary:

• Xen missing feature of PCI group device assignment
• libvirt bug in tracking which PCI device is used where (libvirt.libvirtError: Requested operation is not valid: PCI device 0000:00:1a.0 is in use by driver xenlight, domain usbvm while starting usbvm)

[marmarek]

Summary:

• Xen missing feature of PCI group device assignment
• libvirt bug in tracking which PCI device is used where (libvirt.libvirtError: Requested operation is not valid: PCI device 0000:00:1a.0 is in use by driver xenlight, domain usbvm while starting usbvm)

[Rudd-O]

Quick update: I assigned two of my USB PCI devices (out of three) to a USB VM. That caused hangs and reboots to happen around once each day. They stopped happening as soon as I decided never to power on that VM again. I still have yet to try adding all three USB PCI devices to the USB VM (doing so would disable all USB ports on this machine) We'll see if that causes hangs.

[Rudd-O]

Quick update: I assigned two of my USB PCI devices (out of three) to a USB VM. That caused hangs and reboots to happen around once each day. They stopped happening as soon as I decided never to power on that VM again. I still have yet to try adding all three USB PCI devices to the USB VM (doing so would disable all USB ports on this machine) We'll see if that causes hangs.

[andrewclausen]

The pci_strictreset option didn't have any effect for me. (Exactly the same error messages, etc.)

[andrewclausen]

The pci_strictreset option didn't have any effect for me. (Exactly the same error messages, etc.)

[Rudd-O]

I found my problem. It was a mouse whose receiver stopped working properly, and started causing lockups and hard reboots whenever it was plugged, irrespective of which VMs it was assigned to. The mouse is now in the trash. PCI strict reset did work for starting the VM though.

[Rudd-O]

I found my problem. It was a mouse whose receiver stopped working properly, and started causing lockups and hard reboots whenever it was plugged, irrespective of which VMs it was assigned to. The mouse is now in the trash. PCI strict reset did work for starting the VM though.

[marmarek]

There is a strange issue related to some Logitech receivers: #1689 . I can confirm it indeed happens, but no idea why. I'd rather blame some kernel driver, not the device itself.

[marmarek]

There is a strange issue related to some Logitech receivers: #1689 . I can confirm it indeed happens, but no idea why. I'd rather blame some kernel driver, not the device itself.

[nothingmuch]

I am getting this error too, with pci_strictreset set to false, on a clean install of R3.2 on a Lenovo Yoga 12 which previously had R3.1 working with a usbvm. Disabling USB3 in the bios seemed to work, upgrading the BIOS as mentioned in this thread https://groups.google.com/forum/#!msg/qubes-users/Z6bEMZTjiz4/FbV6T-l_AQAJ did not seem to make a difference.

[nothingmuch]

I am getting this error too, with pci_strictreset set to false, on a clean install of R3.2 on a Lenovo Yoga 12 which previously had R3.1 working with a usbvm. Disabling USB3 in the bios seemed to work, upgrading the BIOS as mentioned in this thread https://groups.google.com/forum/#!msg/qubes-users/Z6bEMZTjiz4/FbV6T-l_AQAJ did not seem to make a difference.

[Rudd-O]

The Logitech device issue is no longer a problem in modern kernels.

[Rudd-O]

The Logitech device issue is no longer a problem in modern kernels.

[nothingmuch]

I'm seeing this with no external USB devices connected.

[nothingmuch]

I'm seeing this with no external USB devices connected.

[Rudd-O]

Since today, I cannot boot the VMs that have USB devices connected to them. The VM doesn't even show an XL console.

[Rudd-O]

Since today, I cannot boot the VMs that have USB devices connected to them. The VM doesn't even show an XL console.

[Rudd-O]

I have pci_strictreset set to false.

[Rudd-O]

I have pci_strictreset set to false.

[marmarek]

Is that after upgrading Xen to 4.6.3?
What is exact error? If you start it from command line, does it show up
in xl list (when qvm-start is still running)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Is that after upgrading Xen to 4.6.3?
What is exact error? If you start it from command line, does it show up
in xl list (when qvm-start is still running)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Rudd-O]

No xl list entry, cannot even open the xen console to it — it just hangs for a little while and then exits.

This single line appears in the xl dmesg:

(XEN) [VT-D] It's risky to assign <PCI device ID> with shared RMRR at <address> for DomNNN

Note that this worked fine before.

[Rudd-O]

No xl list entry, cannot even open the xen console to it — it just hangs for a little while and then exits.

This single line appears in the xl dmesg:

(XEN) [VT-D] It's risky to assign <PCI device ID> with shared RMRR at <address> for DomNNN

Note that this worked fine before.

[Rudd-O]

Correction, the entry appears briefly:

usbvm            437   200   1   ---sc-     0.0

[Rudd-O]

Correction, the entry appears briefly:

usbvm            437   200   1   ---sc-     0.0

[Rudd-O]

qrexec log for that VM says

domain dead
cannot connect to qrexec agent: No such process

[Rudd-O]

qrexec log for that VM says

domain dead
cannot connect to qrexec agent: No such process

[Rudd-O]

libxl-driver.log says:

<time and date> libxl: error: libxl_device.c:1215:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/<domid>0 not ready

[Rudd-O]

libxl-driver.log says:

<time and date> libxl: error: libxl_device.c:1215:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/<domid>0 not ready

[Rudd-O]

Xen on the dom0 is 4.6.1-20.fc23

[Rudd-O]

Xen on the dom0 is 4.6.1-20.fc23

[marmarek]

Probably well known memory fragmentation issue - PV VM with PCI device needs few megs of physically continuous memory for DMA purpose. You can free some by getting it away from dom0: xl mem-set 0 <some-memory-size-in-MB>, where the size is smaller than the current one (for example 500MB smaller). If it does not help, try shutdown some VMs. If still nothing, reboot...

[marmarek]

Probably well known memory fragmentation issue - PV VM with PCI device needs few megs of physically continuous memory for DMA purpose. You can free some by getting it away from dom0: xl mem-set 0 <some-memory-size-in-MB>, where the size is smaller than the current one (for example 500MB smaller). If it does not help, try shutdown some VMs. If still nothing, reboot...

[Rudd-O]

Let me try. But if this works, this really should be documented somewhere!

[Rudd-O]

Let me try. But if this works, this really should be documented somewhere!

[Rudd-O]

Nope, it did not work at all.

[Rudd-O]

Nope, it did not work at all.

[marmarek]

Ok, lets try harder: touch /var/run/qubes/do-not-membalance. Then try again xl mem-set and qvm-start. And if it doesn't work, repeat (just one more time).

[marmarek]

Ok, lets try harder: touch /var/run/qubes/do-not-membalance. Then try again xl mem-set and qvm-start. And if it doesn't work, repeat (just one more time).

[Rudd-O]

On 10/28/2016 10:40 PM, Marek Marczykowski-Górecki wrote:

Ok, lets try harder: |touch /var/run/qubes/do-not-membalance|. Then
try again |xl mem-set| and |qvm-start|. And if it doesn't work, repeat
(just one more time).

A reboot fixed it.

Rudd-O
http://rudd-o.com/

[Rudd-O]

On 10/28/2016 10:40 PM, Marek Marczykowski-Górecki wrote:

Ok, lets try harder: |touch /var/run/qubes/do-not-membalance|. Then
try again |xl mem-set| and |qvm-start|. And if it doesn't work, repeat
(just one more time).

A reboot fixed it.

Rudd-O
http://rudd-o.com/

[xloem]

Same experience. I needed to to touch /var/run/qubes/do-not-membalance to get xl mem-set to do anything at all. I kept dropping the dom0 ram in 512MB increments, and qvm-start kept failing, until the system stopped responding. Then things worked after reboot.

Maybe some file to review to determine memory fragmentation, and where the VM memory is getting allocated, for next time? Or some way to determine what made the VM crash?

[xloem]

Same experience. I needed to to touch /var/run/qubes/do-not-membalance to get xl mem-set to do anything at all. I kept dropping the dom0 ram in 512MB increments, and qvm-start kept failing, until the system stopped responding. Then things worked after reboot.

Maybe some file to review to determine memory fragmentation, and where the VM memory is getting allocated, for next time? Or some way to determine what made the VM crash?