Qubes VM Manager provides out-of-date information on devices

[nvesely]

The block device icon will sometimes stay on VM entries even after the device is detached or shutdown. Also, the attach/detach block entries menu entry will sometimes display out of date information as well.

I interchangeably use the GUI and the CL. This shouldn't be an issue, but I think that's what's causing this problem.

Edit: R3-rc1

[marmarek]

Do you detach device from a VM before physically disconnecting it or shutting down a VM?

[marmarek]

Do you detach device from a VM before physically disconnecting it or shutting down a VM?

[nvesely]

I think I did not. Anyway, I'll do some testing an maybe catch some other problems because QM and the USB passthrough seems a little buggy in general.

First, I'm now using the unstable branch. I'm not sure if this means a new version of Qubes Manager, but perhaps other packages upgrades will have affected the following testing I did.

• Insert USB, MSDOS, single FAT32 partition
• In QM, have the option to attach dom0:sdb (but not dom0:sdb1) as virtual block device to other VMs
• Reformat USB as MSDOS, 2 FAT32 partitions
• QM still sees it as dom0:sdb
• Eject and re-insert, still seen as dom0:sdb
• From dom0 CL, attach dom0:sdb1 to one VM1 and dom0:sdb2 to VM2
• Now in QM I still have option dom0:sdb to any VM (wouldn't this potentially cause problems if I actually did so as multiple OS are allowed to access the same data simultaneously--I guess I'll have to test this). Also, the icons show on VM1 and VM2 as they should after a short delay
• Assign dom0:sdb to VM3 with GUI. Icon immediately shows up.
• Mount /dev/xvdi1 in VM3. Create a file in it.
• Mount /dev/xvdi in VM1. See the file in it.
• After this point, the changes are no longer recognized between the VMs. Create some files in each and wait until later to figure out what happens.
• Mount /dev/xvdi in VM2
• fdisk /dev/xvdi in VM3, delete /dev/xvdi2 and then recreate it (format to FAT32). Fdisk reports it's been altered, but re-reading the partition table fails because device is busy.
• Create some files in VM2 mount point.
• Rip USB out
• All GUI icons disappear
• Plug it back in
• All GUI icons reappear and in QM it's as if it never happened
• lsblk stalls forever in VMs 1 and 2. /dev/xvdi* doesn't exist in VM3
• Detach from VM1 with QM--libvirt error--libxenlight failed to detach disk xvdi at line 1676 of main.py
• QM freezes and must be killed
• Restart QM
• CL and QM can't seem to detach these devices. Both getting libxenlight errors. Shutdown VM 2 and 3. QM freezes up again, this time without warning, and must be killed.
• Turns out ultimately VM 1 and 2 decided which files actually made it onto the VM.

At this point, it's clear I could learn to use a lot more about what's actually going on and why this makes sense given the way Xen makes virtual block devices. dom0, VM1, and QM are still confused--in different ways--as to what's going on and what is where. As far as my bug report goes, I'm not sure how much of this is fixable, and how much needs to be fixed. In Qubes we assume the user is a pretty smart person, but at the same time, we want to protect them from inconvenience and making mistakes that may compromise their security or data integrity. At this point in time, if I remove a USB stick by hand without first detaching it, I have to restart my computer in order for things to work and for the computer to actually realize what's attached. Also, it seems it's possible to be modifying and accessing data from two VMs simultaneously. We can't protect the user from all stupid mistakes, but perhaps, if possible, we can autodetach the virtual block devices if the USB is removed from USBVM and we can put some safeguards to prevent simultaneous access to the same physical address space between two VMs.

I wish this response could have been more cohesive. Apologies.

[nvesely]

I think I did not. Anyway, I'll do some testing an maybe catch some other problems because QM and the USB passthrough seems a little buggy in general.

First, I'm now using the unstable branch. I'm not sure if this means a new version of Qubes Manager, but perhaps other packages upgrades will have affected the following testing I did.

• Insert USB, MSDOS, single FAT32 partition
• In QM, have the option to attach dom0:sdb (but not dom0:sdb1) as virtual block device to other VMs
• Reformat USB as MSDOS, 2 FAT32 partitions
• QM still sees it as dom0:sdb
• Eject and re-insert, still seen as dom0:sdb
• From dom0 CL, attach dom0:sdb1 to one VM1 and dom0:sdb2 to VM2
• Now in QM I still have option dom0:sdb to any VM (wouldn't this potentially cause problems if I actually did so as multiple OS are allowed to access the same data simultaneously--I guess I'll have to test this). Also, the icons show on VM1 and VM2 as they should after a short delay
• Assign dom0:sdb to VM3 with GUI. Icon immediately shows up.
• Mount /dev/xvdi1 in VM3. Create a file in it.
• Mount /dev/xvdi in VM1. See the file in it.
• After this point, the changes are no longer recognized between the VMs. Create some files in each and wait until later to figure out what happens.
• Mount /dev/xvdi in VM2
• fdisk /dev/xvdi in VM3, delete /dev/xvdi2 and then recreate it (format to FAT32). Fdisk reports it's been altered, but re-reading the partition table fails because device is busy.
• Create some files in VM2 mount point.
• Rip USB out
• All GUI icons disappear
• Plug it back in
• All GUI icons reappear and in QM it's as if it never happened
• lsblk stalls forever in VMs 1 and 2. /dev/xvdi* doesn't exist in VM3
• Detach from VM1 with QM--libvirt error--libxenlight failed to detach disk xvdi at line 1676 of main.py
• QM freezes and must be killed
• Restart QM
• CL and QM can't seem to detach these devices. Both getting libxenlight errors. Shutdown VM 2 and 3. QM freezes up again, this time without warning, and must be killed.
• Turns out ultimately VM 1 and 2 decided which files actually made it onto the VM.

At this point, it's clear I could learn to use a lot more about what's actually going on and why this makes sense given the way Xen makes virtual block devices. dom0, VM1, and QM are still confused--in different ways--as to what's going on and what is where. As far as my bug report goes, I'm not sure how much of this is fixable, and how much needs to be fixed. In Qubes we assume the user is a pretty smart person, but at the same time, we want to protect them from inconvenience and making mistakes that may compromise their security or data integrity. At this point in time, if I remove a USB stick by hand without first detaching it, I have to restart my computer in order for things to work and for the computer to actually realize what's attached. Also, it seems it's possible to be modifying and accessing data from two VMs simultaneously. We can't protect the user from all stupid mistakes, but perhaps, if possible, we can autodetach the virtual block devices if the USB is removed from USBVM and we can put some safeguards to prevent simultaneous access to the same physical address space between two VMs.

I wish this response could have been more cohesive. Apologies.

[marmarek]

First of all, thanks for thorough test report! :)

Few things I think will clarify a lot here:

1. Qubes Manager intentionally does not allow to attach individual
   partitions to the VMs. Details here:
   #623

   The side effect of this decision is also making hard to attach the
   same device to multiple VMs simultaneously (the whole device and its
   partition). @bnvk, do you think we should change this?

2. Current block device framework does not provide an information that
   two devices shouldn't be used at the same time. We can't rely on
   device name (dom0:sdb and dom0:sdb1), because it is just opaque string
   provided by VM to which this device is physically connected (can be some
   USB VM or sth like this). We might introduce such a feature, but it
   is really low prio (create a separate issue if you think it worth any
   effort).

3. Currently (R3.0) device must be detached from VM before physically
   unplugging it. Xen toolstack used in R3.0 (libxl + libvirt) does not
   have any device monitoring mechanisms. If the device is physically
   removed, the toolstack will still think the device is present and
   connected (also some settings of xen block frontend and backend will not
   be cleaned up because of that). The only way to tell the toolstack that
   device is no longer connected, is to detach it from VM (action initiated
   by the user, through qubes manager or qvm-block, not by backend driver
   in response to disconnecting the device). But if the device is no longer
   there, such detach action would fail. There is a way to manually clean
   that mess (basically reconnect the device using low level commands, and
   then properly detach it), but it isn't anything we want to use in any
   way.

   In R2 we had a nasty hack for that, because toolstack in 4.1 was even
   more primitive in this manner (it hadn't track on connected devices).
   But it isn't applicable to R3.

   The proper fix would be to implement device monitoring facilities in
   libxl, then use that API in libvirt. But this a lot work and probably
   won't happen anytime soon...

I think none of above is documented anywhere. Where would you expect to
find such info? I'll paste it there.

[marmarek]

First of all, thanks for thorough test report! :)

Few things I think will clarify a lot here:

1. Qubes Manager intentionally does not allow to attach individual
   partitions to the VMs. Details here:
   #623

   The side effect of this decision is also making hard to attach the
   same device to multiple VMs simultaneously (the whole device and its
   partition). @bnvk, do you think we should change this?

2. Current block device framework does not provide an information that
   two devices shouldn't be used at the same time. We can't rely on
   device name (dom0:sdb and dom0:sdb1), because it is just opaque string
   provided by VM to which this device is physically connected (can be some
   USB VM or sth like this). We might introduce such a feature, but it
   is really low prio (create a separate issue if you think it worth any
   effort).

3. Currently (R3.0) device must be detached from VM before physically
   unplugging it. Xen toolstack used in R3.0 (libxl + libvirt) does not
   have any device monitoring mechanisms. If the device is physically
   removed, the toolstack will still think the device is present and
   connected (also some settings of xen block frontend and backend will not
   be cleaned up because of that). The only way to tell the toolstack that
   device is no longer connected, is to detach it from VM (action initiated
   by the user, through qubes manager or qvm-block, not by backend driver
   in response to disconnecting the device). But if the device is no longer
   there, such detach action would fail. There is a way to manually clean
   that mess (basically reconnect the device using low level commands, and
   then properly detach it), but it isn't anything we want to use in any
   way.

   In R2 we had a nasty hack for that, because toolstack in 4.1 was even
   more primitive in this manner (it hadn't track on connected devices).
   But it isn't applicable to R3.

   The proper fix would be to implement device monitoring facilities in
   libxl, then use that API in libvirt. But this a lot work and probably
   won't happen anytime soon...

I think none of above is documented anywhere. Where would you expect to
find such info? I'll paste it there.

[nvesely]

In reply to your points:

1. In my experience attaching a single partition did show up in nautilus. Maybe reconsider allowing this granularity since this is now the case?
2. I think it is something to keep an eye on. Low-priority though sounds right.
3. Same comments as 2.

I think none of above is documented anywhere. Where would you expect to find such info? I'll paste it there.

Probably here https://www.qubes-os.org/doc/StickMounting/.

There is a way to manually clean that mess (basically reconnect the device using low level commands, and then properly detach it), but it isn't anything we want to use in any way.

In case someone makes the mistake of physically removing a device or shutting down a VM before detaching from it, it would be nice to know how to get things working again w/o having to reboot. Sometimes rebooting is not an option while you have an ongoing process and you also need to have full PVUSB (is this the right word for the technology used to attach virtual block devices to other VMs from USBVM?) support.

[nvesely]

In reply to your points:

1. In my experience attaching a single partition did show up in nautilus. Maybe reconsider allowing this granularity since this is now the case?
2. I think it is something to keep an eye on. Low-priority though sounds right.
3. Same comments as 2.

I think none of above is documented anywhere. Where would you expect to find such info? I'll paste it there.

Probably here https://www.qubes-os.org/doc/StickMounting/.

There is a way to manually clean that mess (basically reconnect the device using low level commands, and then properly detach it), but it isn't anything we want to use in any way.

In case someone makes the mistake of physically removing a device or shutting down a VM before detaching from it, it would be nice to know how to get things working again w/o having to reboot. Sometimes rebooting is not an option while you have an ongoing process and you also need to have full PVUSB (is this the right word for the technology used to attach virtual block devices to other VMs from USBVM?) support.

[nvesely]

Should I make one regarding point 3 as well or is this too hypothetical/ dependent on some not-even-in-the-works libxl facilities?

[nvesely]

Should I make one regarding point 3 as well or is this too hypothetical/ dependent on some not-even-in-the-works libxl facilities?

[marmarek]

On Thu, Jul 23, 2015 at 04:48:31PM -0700, Noah Vesely wrote:

Should I make one regarding point 3 as well or is this too hypothetical/ dependent on some not-even-in-the-works libxl facilities?

You can make one, we can refer to it in documentation so it will be
simpler to notice when the feature is actually implemented.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Thu, Jul 23, 2015 at 04:48:31PM -0700, Noah Vesely wrote:

Should I make one regarding point 3 as well or is this too hypothetical/ dependent on some not-even-in-the-works libxl facilities?

You can make one, we can refer to it in documentation so it will be
simpler to notice when the feature is actually implemented.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[nvesely]

You can make one, we can refer to it in documentation so it will be simpler to notice when the feature is actually implemented.

I went ahead and made one. Should definitely be explained in the docs though.

[nvesely]

You can make one, we can refer to it in documentation so it will be simpler to notice when the feature is actually implemented.

I went ahead and made one. Should definitely be explained in the docs though.

[marmarek]

Ok, I think all the issue(s) here either separated to other tickets or fixed (do not consider device attached when VM is shut down).

[marmarek]

Ok, I think all the issue(s) here either separated to other tickets or fixed (do not consider device attached when VM is shut down).