sys-usb works abnormally, attached usb drives do not show and cant be removed unless VM Manager is crashed (Houston..)

[jamog]

Qubes OS version:

R3.2.

Affected component(s):

sys-usb, app VMs, external drives

---

Steps to reproduce the behavior:

start an app VM. insert an external drive. attach the usb to the VM.
then try to remove the usb.

Expected behavior:

open file manager, terminal or gnome disks, and see the usb there, and operate on it.
you detach the usb by right-clicking on app VM in Qubes VM Manager, and it is detached.

Actual behavior:

after inserting usb in the machine;

1. the usb is mounted, you see and operate on it, and anywhere between 5 seconds and 5 minutes (most often within a minute) the popup message appears that says the usb is detached.
2. the usb is shown attached, and in the next second it is detached again (message windows popping up one after another)
3. the usb is shown attached, but you don't see it anywhere, whether in file manager, terminal or gnome disks.
4. and, unlike until a short while ago, the usb requires manual mounting and it is then mounted under '/mnt', whereas it used to automatically get mounted under '/media/user/somename' and hence be immediately visible in file manager.

when trying to remove the usb from the VM;

1. usb cannot be removed.
2. popup window appears saying 'libvirtError: internal error: libxenlight failed to detach disk 'xvdi'
   This is most likely a bug in the Qubes Manager'.
   this error message window cannot be closed. when you try several times by clicking on the upper right corner, a new window appears saying the application might be busy and asking me if I want to close the window. choosing 'yes', the Qubes VM Manager window closes.
   and when I restart the VM Manager, I cannot do anything with the usb anymore. whether attached. detached, removed, re-inserted, the system only attaches it and does nothing more. if re-inserted, you need to go through the same things I described.
3. on my fresh installed system, in one case so far, one app VM was shutdown but the usb icon was still shown on it. when I restarted this app VM, the icon disappeared.

as for sys-usb:
before I reinstalled Qubes, in the old system, sys-usb was often not started with the launch of the desktop, either. I'd thought this and the above other problems might have had to do with my manual, later installation of sys-usb instead of configuring it right after the Qubes installation. but it proved that it didn't.
in the current fresh install I configured Qubes with sys-usb creation, but the moment I inserted the first usb, all problems were there as before, except maybe the sys-usb didn't fail to launch at the start of desktop session yet.

while in the old installation restarting sys-usb could make the inserted usb drives visible, thereby mountable and operable "momentarily", in the new one even this doesnt work. instead, in one session, sys-usb didn't start anymore after stopping it, with the error that reported being unable to run qrexec-daemon.

General notes:

obviously there is a problem with sys-usb. I didn't have it for a long time in my old system, and never had problems with external drives. after I installed it, I still didn't have any issues until lately.

---

Related issues:

before I reinstalled Qubes, it also happened several times that, following these problematic sessions with usb drives, the shutdown process also took Qubes so much longer than usual that I had to manually power it off.
not to mention sys-whonix, which lately and often failed to start at the beginning of a desktop session, and reinstalling it didn't help solve this.
another strange phenomenon is that in the old installation the machine was twice started in the middle of the night while it was suspended with the lid closed. this was the first one of all the unusual behaviour outlined above.

[lunarthegrey]

sys-usb was strange in 3.2 for me as well. But in 4.0 it seems to be fine so far. How much memory are you allowing sys-usb? I had several issues because I wasn't giving it enough memory and it would crash. What template are you using for sys-usb as well? Is it up to date?

[lunarthegrey]

sys-usb was strange in 3.2 for me as well. But in 4.0 it seems to be fine so far. How much memory are you allowing sys-usb? I had several issues because I wasn't giving it enough memory and it would crash. What template are you using for sys-usb as well? Is it up to date?

[jamog]

I am using fedora-26 for sys-usb and it is up-to-date. the memory has 3GB max. memory.

after posting about the problem I was able to use the external drive for several hours without interruption - that is not to say in normal pace; it was still being detached right after attachment, and yet remaining visible and operable on via file manager.
likewise, the problem with removal is still there, too. the aforementioned 'libxenlight..' error window followed by 'Houston..wanna terminate the application?' window and the closure of VM Manager.

so, as it currently seems, somehow, once the usb is inserted, the data saved on it is accessible and usable, but the usb keeps showing the outlined array of crazy behaviour.

[jamog]

I am using fedora-26 for sys-usb and it is up-to-date. the memory has 3GB max. memory.

after posting about the problem I was able to use the external drive for several hours without interruption - that is not to say in normal pace; it was still being detached right after attachment, and yet remaining visible and operable on via file manager.
likewise, the problem with removal is still there, too. the aforementioned 'libxenlight..' error window followed by 'Houston..wanna terminate the application?' window and the closure of VM Manager.

so, as it currently seems, somehow, once the usb is inserted, the data saved on it is accessible and usable, but the usb keeps showing the outlined array of crazy behaviour.

[lunarthegrey]

Hmm attach these logs here after you notice an issue, especially after the USB gets randomly detached.

/var/log/xen/console/guest-sys-usb.log
/var/log/qubes/qrexec.sys-usb.log

I'm not 100% sure that these are the right logs, as I've never had such a strange issue myself. @marmarek might be able to point you in the right direction if I'm wrong.

[lunarthegrey]

Hmm attach these logs here after you notice an issue, especially after the USB gets randomly detached.

/var/log/xen/console/guest-sys-usb.log
/var/log/qubes/qrexec.sys-usb.log

I'm not 100% sure that these are the right logs, as I've never had such a strange issue myself. @marmarek might be able to point you in the right direction if I'm wrong.

[Polygonbugs]

I can reproduce this issue. I'm not sure it is identical case of @jamog but surely I can. And I think this is duplicate issue since I think someone already reported this problem.

The way to reproduce problem

1. Attach two USB devices in one USB memory controller(pci) assigned VM(which is usually Sys-USB).
2. Attach two USB in one VM other than Sys-USB via either command line or Qubes-Manager.
3. Try to detach one of USB devices via either command line or Qubes-Manger from the VM.
4. Qubes Manger crashes with houston error and in commandline shows 'libvirtError: internal error: libxenlight failed to detach disk 'xvdi'.
5. Try to remove USB with physical way.
6. Try to insert other USB sticks on the same VM or another VM.
7. Qubes Manager shows that it is inserted but the device is not really attached.

Workaround of this problem
Remove all USB by using physical way. Then reboot Sys-USB.

Information
I'm using Qubes-OS 3.2 and using fedora-26-minimal as template of Sys-USB.

---

I am using fedora-26 for sys-usb and it is up-to-date. the memory has 3GB max. memory.

It is waste of memory. Sys-USB won't take much memory. See memory usage of the Sys-USB via 'top' command.

before I reinstalled Qubes, it also happened several times that, following these problematic sessions with usb drives, the shutdown process also took Qubes so much longer than usual that I had to manually power it off.

Before you shutting down system, shutdown or kill all domUs. Then system close fast. For me it takes 5~6 seconds? I use "poweroff" command in dom0.

not to mention sys-whonix, which lately and often failed to start at the beginning of a desktop session, and reinstalling it didn't help solve this.

I'm not sure this issue is related to Sys-USB. Rather, it is more related to after patch of Meltdown & Spectre. Especially, after implementing XPTI. At least for me...

another strange phenomenon is that in the old installation the machine was twice started in the middle of the night while it was suspended with the lid closed. this was the first one of all the unusual behaviour outlined above.

Do you mean by VM? Then maybe OOM-killer activated? This is another not fixed issue. This also could be applied Qubes-OS 4.0. You can see related issue at #3079. If it is related to physical machine, I doubt that you enabled Power On Lan?

[Polygonbugs]

I can reproduce this issue. I'm not sure it is identical case of @jamog but surely I can. And I think this is duplicate issue since I think someone already reported this problem.

The way to reproduce problem

1. Attach two USB devices in one USB memory controller(pci) assigned VM(which is usually Sys-USB).
2. Attach two USB in one VM other than Sys-USB via either command line or Qubes-Manager.
3. Try to detach one of USB devices via either command line or Qubes-Manger from the VM.
4. Qubes Manger crashes with houston error and in commandline shows 'libvirtError: internal error: libxenlight failed to detach disk 'xvdi'.
5. Try to remove USB with physical way.
6. Try to insert other USB sticks on the same VM or another VM.
7. Qubes Manager shows that it is inserted but the device is not really attached.

Workaround of this problem
Remove all USB by using physical way. Then reboot Sys-USB.

Information
I'm using Qubes-OS 3.2 and using fedora-26-minimal as template of Sys-USB.

---

I am using fedora-26 for sys-usb and it is up-to-date. the memory has 3GB max. memory.

It is waste of memory. Sys-USB won't take much memory. See memory usage of the Sys-USB via 'top' command.

before I reinstalled Qubes, it also happened several times that, following these problematic sessions with usb drives, the shutdown process also took Qubes so much longer than usual that I had to manually power it off.

Before you shutting down system, shutdown or kill all domUs. Then system close fast. For me it takes 5~6 seconds? I use "poweroff" command in dom0.

not to mention sys-whonix, which lately and often failed to start at the beginning of a desktop session, and reinstalling it didn't help solve this.

I'm not sure this issue is related to Sys-USB. Rather, it is more related to after patch of Meltdown & Spectre. Especially, after implementing XPTI. At least for me...

another strange phenomenon is that in the old installation the machine was twice started in the middle of the night while it was suspended with the lid closed. this was the first one of all the unusual behaviour outlined above.

Do you mean by VM? Then maybe OOM-killer activated? This is another not fixed issue. This also could be applied Qubes-OS 4.0. You can see related issue at #3079. If it is related to physical machine, I doubt that you enabled Power On Lan?

[jamog]

@Polygonbugs thank you for your tips. meanwhile I've changed the memory settings.
yes, I shutdown VMs before turning off the machine.
I've switched to 4.0. meanwhile, so far sys-whonix problem doesn't occur anymore.
also the machine has not restarted again when the lid was closed since.

AND..still, with 4.0., too, the usb devices keep getting disconnected and connected back - though not each and every usb device connected, it keeps happening and being a disturbance.

as for the problem with usb removal, I seem to have observed that Qubes Manager definitely enters into "Houston" state or some sort of disorder when the usb device is not unmounted first before it is disconnected from the VM.

[jamog]

@Polygonbugs thank you for your tips. meanwhile I've changed the memory settings.
yes, I shutdown VMs before turning off the machine.
I've switched to 4.0. meanwhile, so far sys-whonix problem doesn't occur anymore.
also the machine has not restarted again when the lid was closed since.

AND..still, with 4.0., too, the usb devices keep getting disconnected and connected back - though not each and every usb device connected, it keeps happening and being a disturbance.

as for the problem with usb removal, I seem to have observed that Qubes Manager definitely enters into "Houston" state or some sort of disorder when the usb device is not unmounted first before it is disconnected from the VM.