qvm-block -A fails without breaking

[GammaSQ]

Qubes OS version R3.1:

This affects the file-mounting capabilities of qvm-block within dom0

---

Expected behavior:

qvm-block -A [vm] [backend]:[/path/to/file] should throw an error and exit if anything goes wrong.

Actual behavior:

qvm-block -A exits without any trouble, qvm-block -l claims the file was attached as xvd[x], though the VM does not show a corresponding /dev/xvd[x] node. This seems to happen when the VM does not have enough RAM (not sure where the threshold is, testing is quite time consuming. Failed VMs had >1GB)
qvm-block -d [vm] throws an error:

qvm-block -d sys-net
Traceback (most recent call last):
  File "/usr/bin/qvm-block", line 151, in <module>
    main()
  File "/usr/bin/qvm-block", line 122, in main
    block_detach_all(vm)
  File "/usr/lib64/python2.7/site-packages/qubes/qubesutils.py", line 459, in block_detach_all
    block_detach(vm, None)
  File "/usr/lib64/python2.7/site-packages/qubes/qubesutils.py", line 447, in block_detach
    vm.libvirt_domain.detachDevice(etree.tostring(disk, encoding='utf-8'))
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1146, in detachDevice
    if ret == -1: raise libvirtError ('virDomainDetachDevice() failed', dom=self)
libvirt.libvirtError: internal error: libxenlight failed to detach disk 'xvdi'

(obviously, /dev/xvdi never existed in the first place!)

Steps to reproduce the behavior:

create a virtual thumbdrive within the VM:
dd if=/dev/zero of=~/thumb.img bs=1M count=10000
losetup --partscan --find thumb.img
mkfs.ext4 /dev/loop0
(perhaps confirm that mount /dev/loop0 /mnt works, but don't forget to umount!)

in dom0, use the qvm-block cmd:

qvm-block -A [BIGVM] [sourceVM]:/home/user/tumb.img     #should work
qvm-block -d                                                                        #should still work
qvm-block -A [smallvm] [sourceVM]:/home/user/thumb.img #whould break, but will work
[confirm /dev/xvd[x] doesn't exit in VM]
qvm-block                                                                            #shows "attached" device
qvm-block -d                                                                        #crashes

General notes:

I'm using Kernel 4.4.10-9.pvops.qubes.x86_64 from the unstable repo.
I've tried my luck with xl block-attach and block-detach, however, block-detach ALWAYS failes due to error:

libxl: error: libxl_device.c:301:libxl__device_disk_set_backend: no suitable backend for disk xvdi
libxl_device_disk_remove_failed.

which probably has something to do with this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=912488

---

Related issues:

probably: #2018

[GammaSQ]

Some additional information:
The Memory-limit seems to be ~400MB maximum memory.
Valid for VMs of all sizes:
qvm-block -l does not display the expected "attached to 'vault' as 'xvdi'" information, but just:
backup:loop0 /home/user/backup.img 9 GiB
Once it's attached to one VM, it can be easily reattached to another one using the Qubes VM-Manager, and then the expected message appears:
backup:loop0 /home/user/backup.img 9 GiB (attached to 'vault' as 'xvdi')
However, the QVM-Manager fails to detach the device from the previousely attached VM.

[GammaSQ]

Some additional information:
The Memory-limit seems to be ~400MB maximum memory.
Valid for VMs of all sizes:
qvm-block -l does not display the expected "attached to 'vault' as 'xvdi'" information, but just:
backup:loop0 /home/user/backup.img 9 GiB
Once it's attached to one VM, it can be easily reattached to another one using the Qubes VM-Manager, and then the expected message appears:
backup:loop0 /home/user/backup.img 9 GiB (attached to 'vault' as 'xvdi')
However, the QVM-Manager fails to detach the device from the previousely attached VM.

[andrewdavidwong]

I can reproduce this on 3.2. It completely blocks upgrading StandaloneVMs (and, I assume, TemplateVMs) per our instructions.

[andrewdavidwong]

I can reproduce this on 3.2. It completely blocks upgrading StandaloneVMs (and, I assume, TemplateVMs) per our instructions.

[marmarek]

@andrewdavidwong according to the report, it affect only VMs with little memory. Is it your case, or maybe it affect also others?

[marmarek]

@andrewdavidwong according to the report, it affect only VMs with little memory. Is it your case, or maybe it affect also others?

[marmarek]

@andrewdavidwong ok, I reproduced it, but I think this is a different issue: qvm-block -A do attach the device (/dev/xvdi appears in the VM), but later qvm-block -l do not list it and it's impossible to detach it using standard Qubes tools. Shouldn't affect upgrade, as it rely on automatic detach on VM shutdown (which does work).

[marmarek]

@andrewdavidwong ok, I reproduced it, but I think this is a different issue: qvm-block -A do attach the device (/dev/xvdi appears in the VM), but later qvm-block -l do not list it and it's impossible to detach it using standard Qubes tools. Shouldn't affect upgrade, as it rely on automatic detach on VM shutdown (which does work).

[marmarek]

@GammaSQ I can't reproduce it with a VM running kernel 4.4.31. Even VM with maxmem 200M works as expected. qvm-block -l do not list such device, but as explained, it is separate issue.

[marmarek]

@GammaSQ I can't reproduce it with a VM running kernel 4.4.31. Even VM with maxmem 200M works as expected. qvm-block -l do not list such device, but as explained, it is separate issue.

[andrewdavidwong]

ok, I reproduced it, but I think this is a different issue: qvm-block -A do attach the device (/dev/xvdi appears in the VM), but later qvm-block -l do not list it and it's impossible to detach it using standard Qubes tools. Shouldn't affect upgrade, as it rely on automatic detach on VM shutdown (which does work).

Yes, you're right.

[andrewdavidwong]

ok, I reproduced it, but I think this is a different issue: qvm-block -A do attach the device (/dev/xvdi appears in the VM), but later qvm-block -l do not list it and it's impossible to detach it using standard Qubes tools. Shouldn't affect upgrade, as it rely on automatic detach on VM shutdown (which does work).

Yes, you're right.

[andrewdavidwong]

Just wanted to report that I still experience this issue (the actual one this time) once in a while. I have to restart the VM, then re-attach to get it to work correctly.

[andrewdavidwong]

Just wanted to report that I still experience this issue (the actual one this time) once in a while. I have to restart the VM, then re-attach to get it to work correctly.

[taonik]

In R4.0 rc1 xl block-detach behaves in the same way.
Reboot is required.

[taonik]

In R4.0 rc1 xl block-detach behaves in the same way.
Reboot is required.