USB VM does not recover from "not enough memory"

[bnvk]

I experienced this early while doing a bunch of stuff, so I shut everything down and rebooted my machine to re-test my assumptions. The conditions I encountered are:

• USB VM is not running. Was shutdown intentionally.
• Use of other VMs at a maximum so that there is no memory available
• The Out of of Memorry notification shows upon launching a VM
• Shutdown a few VMs to free up memory
• Launch USB VM and it fails due to no memory
• Then no matter how much memory is free, USB VM can never boot up

Instead of booting up, it shows the error:

Error starting VM: 'usbdata': Cannot execute qrexec-daemon!

I wonder if this is connected to #878 or #1148 ?

[marmarek]

No, it's most likely about: #1038

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

No, it's most likely about: #1038

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[bnvk]

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

@marmarek is that achieved with qvm-prefs -s usbdata kernelopts "nopat iommu=soft swiotlb=soft" ?

This same issue just happened following a different path of behavior:

• Create new VM usbdata-2 try to assign USB driver to it
• See message about driver already being in use by usbdata-1
• Shutdown usbdata-1 VM manually
• Assign driver to usbdata-2
• Neither VM will start up and instead show qrexec-daemon error

[bnvk]

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

@marmarek is that achieved with qvm-prefs -s usbdata kernelopts "nopat iommu=soft swiotlb=soft" ?

This same issue just happened following a different path of behavior:

• Create new VM usbdata-2 try to assign USB driver to it
• See message about driver already being in use by usbdata-1
• Shutdown usbdata-1 VM manually
• Assign driver to usbdata-2
• Neither VM will start up and instead show qrexec-daemon error

[marmarek]

On Sat, Jan 23, 2016 at 10:40:32AM -0800, Brennan Novak wrote:

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

@marmarek is that achieved with qvm-prefs -s usbdata kernelopts "nopat iommu=soft swiotlb=soft" ?

Not swiotlb=soft but swiotlb=2048 (or some even lower value).

This same issue just happened following a different path of behavior:

• Create new VM usbdata-2 try to assign USB driver to it
• See message about driver already being in use by usbdata-1
• Shutdown usbdata-1 VM manually
• Assign driver to usbdata-2
• Neither VM will start up and instead show qrexec-daemon error

Generally it is hard to start a VM with PCI device after some Qubes
runtime, because of memory fragmentation. Some details are described
in #1038. There is a hacky workaround for that:

1. Pause dynamic memory management: touch
   /var/run/qubes/do-not-membalance
2. Get some big chunk of memory from dom0: xl mem-set 0 1500
3. Start the VM: qvm-start usbdata-2
4. Re-enable memory management: rm /var/run/qubes/do-not-membalance

You need to adjust value in step 2 to your needs, often having 1-2GB
free is enough (check free memory with xl info).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Sat, Jan 23, 2016 at 10:40:32AM -0800, Brennan Novak wrote:

You can try to lower swiotlb kernel parameter, but most likely you'll
need to set USB VM autostart and reboot the machine.

@marmarek is that achieved with qvm-prefs -s usbdata kernelopts "nopat iommu=soft swiotlb=soft" ?

Not swiotlb=soft but swiotlb=2048 (or some even lower value).

This same issue just happened following a different path of behavior:

• Create new VM usbdata-2 try to assign USB driver to it
• See message about driver already being in use by usbdata-1
• Shutdown usbdata-1 VM manually
• Assign driver to usbdata-2
• Neither VM will start up and instead show qrexec-daemon error

Generally it is hard to start a VM with PCI device after some Qubes
runtime, because of memory fragmentation. Some details are described
in #1038. There is a hacky workaround for that:

1. Pause dynamic memory management: touch
   /var/run/qubes/do-not-membalance
2. Get some big chunk of memory from dom0: xl mem-set 0 1500
3. Start the VM: qvm-start usbdata-2
4. Re-enable memory management: rm /var/run/qubes/do-not-membalance

You need to adjust value in step 2 to your needs, often having 1-2GB
free is enough (check free memory with xl info).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?