Attaching block devices from dispVM broken #2126

rustybird · Jun 29, 2016

Qubes OS version (e.g., `R3.1`):

R3.1

Affected TemplateVMs (e.g., `fedora-23`, if applicable):

Tested with fedora-23 and fedora-23-minimal dispVMs:

# start a new dispVM:
[dom0] $ echo xterm | /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell dom0 DEFAULT gray

# create a loop device there:
[disp8] $ truncate -s 1G foo
[disp8] # losetup /dev/loop0 foo

# attach it to another vm:
[dom0] $ qvm-block | grep loop0
disp8:loop0 /home/user/foo 1024 MiB
[dom0] $ qvm-block -a testvm disp8:loop0
[dom0] $ qvm-block | grep loop0
disp8:loop0 /home/user/foo 1024 MiB (attached to 'testvm' as 'xvdi')

#####################################################
# But it doesn't show up in testvm's dmesg or /dev. #
#####################################################

# try to detach it:
[dom0] $ qvm-block -d disp8:loop0
Traceback (most recent call last):
  File "/usr/bin/qvm-block", line 151, in <module>
    main()
  File "/usr/bin/qvm-block", line 133, in main
    block_detach(attached_to['vm'], attached_to['frontend'])
  File "/usr/lib64/python2.7/site-packages/qubes/qubesutils.py", line 447, in block_detach
    vm.libvirt_domain.detachDevice(etree.tostring(disk, encoding='utf-8'))
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1146, in detachDevice
    if ret == -1: raise libvirtError ('virDomainDetachDevice() failed', dom=self)
libvirt.libvirtError: internal error: libxenlight failed to detach disk 'xvdi'
[dom0] $ journalctl -b -o short-monotonic
...
[ 8745.585064] dom0 block-cleaner-daemon.py[1481]: libxl: error: libxl_device.c:301:libxl__device_disk_set_backend: no suitable backend for disk xvdi
[ 8745.587866] dom0 block-cleaner-daemon.py[1481]: libxl_device_disk_remove failed.
[ 8755.072768] dom0 libvirtd[1351]: internal error: libxenlight failed to detach disk 'xvdi'
[ 8755.135839] dom0 libvirtd[1351]: End of file while reading data: Input/output error
...

Using xl block-attach instead of qvm-block --attach has the same problem:

# before attaching:
dom0 $ xl block-list testvm2
Vdev  BE  handle state evt-ch ring-ref BE-path                       
51712 0   56     4     31     8        /local/domain/0/backend/vbd/56/51712
51728 0   56     4     32     9        /local/domain/0/backend/vbd/56/51728
51744 0   56     4     33     10       /local/domain/0/backend/vbd/56/51744
51760 0   56     4     34     11       /local/domain/0/backend/vbd/56/51760


dom0 $ sudo xl block-attach testvm2 phy:/dev/loop0 backend=disp20 xvdi w
dom0 $ echo $?
0

dom0 $ xl list disp20
Name                                        ID   Mem VCPUs  State   Time(s)
disp20                                      57   315     1     -b----       9.2

dom0 $ xl block-list testvm2
Vdev  BE  handle state evt-ch ring-ref BE-path                       
51712 0   56     4     31     8        /local/domain/0/backend/vbd/56/51712
51728 0   56     4     32     9        /local/domain/0/backend/vbd/56/51728
51744 0   56     4     33     10       /local/domain/0/backend/vbd/56/51744
51760 0   56     4     34     11       /local/domain/0/backend/vbd/56/51760
51840 57  56     3     47     2638     /local/domain/57/backend/vbd/56/51840

######################################################
# But it doesn't show up in testvm2's dmesg or /dev. #
######################################################


dom0 $ sudo xl block-detach testvm2 51840
libxl: error: libxl_device.c:301:libxl__device_disk_set_backend: no suitable backend for disk xvdi
libxl_device_disk_remove failed.

rustybird · Jun 30, 2016

Using xl block-attach instead of qvm-block --attach has the same problem:

# before attaching:
dom0 $ xl block-list testvm2
Vdev  BE  handle state evt-ch ring-ref BE-path                       
51712 0   56     4     31     8        /local/domain/0/backend/vbd/56/51712
51728 0   56     4     32     9        /local/domain/0/backend/vbd/56/51728
51744 0   56     4     33     10       /local/domain/0/backend/vbd/56/51744
51760 0   56     4     34     11       /local/domain/0/backend/vbd/56/51760


dom0 $ sudo xl block-attach testvm2 phy:/dev/loop0 backend=disp20 xvdi w
dom0 $ echo $?
0

dom0 $ xl list disp20
Name                                        ID   Mem VCPUs  State   Time(s)
disp20                                      57   315     1     -b----       9.2

dom0 $ xl block-list testvm2
Vdev  BE  handle state evt-ch ring-ref BE-path                       
51712 0   56     4     31     8        /local/domain/0/backend/vbd/56/51712
51728 0   56     4     32     9        /local/domain/0/backend/vbd/56/51728
51744 0   56     4     33     10       /local/domain/0/backend/vbd/56/51744
51760 0   56     4     34     11       /local/domain/0/backend/vbd/56/51760
51840 57  56     3     47     2638     /local/domain/57/backend/vbd/56/51840

######################################################
# But it doesn't show up in testvm2's dmesg or /dev. #
######################################################


dom0 $ sudo xl block-detach testvm2 51840
libxl: error: libxl_device.c:301:libxl__device_disk_set_backend: no suitable backend for disk xvdi
libxl_device_disk_remove failed.

The last line from block-list looks interesting: It appears that the disk does not transition from XenbusStateInitialised (= 3) to to XenbusStateConnected (= 4).

rustybird · Jun 30, 2016

The last line from block-list looks interesting: It appears that the disk does not transition from XenbusStateInitialised (= 3) to to XenbusStateConnected (= 4).

Do you see anything interesting in kernel messages of either VM? Or maybe in /var/log/xen/xen-hotplug.log of dispvm?

marmarek · Jun 30, 2016

Do you see anything interesting in kernel messages of either VM? Or maybe in /var/log/xen/xen-hotplug.log of dispvm?

Do you see anything interesting in kernel messages of either VM?

In the destination testvm, nothing at all. But there's something kind of interesting about a disposable source VM vs. a normal (non-disposable, successful) source VM -- using sudo journald here to show both kernel and userspace logs.

disposable source:

[  309.965670] fedora-23-clone-1-dvm kernel: xen-blkback: event-channel 46
[  309.967128] fedora-23-clone-1-dvm kernel: xen-blkback: /local/domain/75/device/vbd/51840:using single page: ring-ref 2946
[  309.968078] fedora-23-clone-1-dvm kernel: xen-blkback: ring-pages:1, event-channel 46, protocol 1 (x86_64-abi) persistent grants

normal source:

[ 7309.744756] normalvm kernel: xen-blkback: event-channel 50
[ 7309.747492] normalvm kernel: xen-blkback: /local/domain/75/device/vbd/51856:using single page: ring-ref 2947
[ 7309.748369] normalvm kernel: xen-blkback: ring-pages:1, event-channel 50, protocol 1 (x86_64-abi) persistent grants
[ 7309.754343] normalvm kernel: xen-blkback: event-channel 50
[ 7309.761667] normalvm kernel: xen-blkback: /local/domain/75/device/vbd/51856:using single page: ring-ref 2947
[ 7309.764652] normalvm kernel: xen-blkback: ring-pages:1, event-channel 50, protocol 1 (x86_64-abi) persistent grants
[ 7309.912848] normalvm root[2435]: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/75/51856
[ 7310.371945] normalvm root[2484]: /etc/xen/scripts/block: Writing backend/vbd/75/51856/physical-device 7:0 to xenstore.
[ 7310.424383] normalvm root[2486]: /etc/xen/scripts/block: Writing backend/vbd/75/51856/hotplug-status connected to xenstore.

So, in a disposable VM, there's three kernel lines. Whereas in the successful case of a non-disposable VM, there's three kernel lines, then the same three lines repeated, then /etc/xen/scripts/block runs.

Or maybe in /var/log/xen/xen-hotplug.log of dispvm?

Doesn't exist, only empty /var/log/xen/xldev.log* files.

rustybird · Jun 30, 2016

Do you see anything interesting in kernel messages of either VM?

In the destination testvm, nothing at all. But there's something kind of interesting about a disposable source VM vs. a normal (non-disposable, successful) source VM -- using sudo journald here to show both kernel and userspace logs.

disposable source:

[  309.965670] fedora-23-clone-1-dvm kernel: xen-blkback: event-channel 46
[  309.967128] fedora-23-clone-1-dvm kernel: xen-blkback: /local/domain/75/device/vbd/51840:using single page: ring-ref 2946
[  309.968078] fedora-23-clone-1-dvm kernel: xen-blkback: ring-pages:1, event-channel 46, protocol 1 (x86_64-abi) persistent grants

normal source:

[ 7309.744756] normalvm kernel: xen-blkback: event-channel 50
[ 7309.747492] normalvm kernel: xen-blkback: /local/domain/75/device/vbd/51856:using single page: ring-ref 2947
[ 7309.748369] normalvm kernel: xen-blkback: ring-pages:1, event-channel 50, protocol 1 (x86_64-abi) persistent grants
[ 7309.754343] normalvm kernel: xen-blkback: event-channel 50
[ 7309.761667] normalvm kernel: xen-blkback: /local/domain/75/device/vbd/51856:using single page: ring-ref 2947
[ 7309.764652] normalvm kernel: xen-blkback: ring-pages:1, event-channel 50, protocol 1 (x86_64-abi) persistent grants
[ 7309.912848] normalvm root[2435]: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/75/51856
[ 7310.371945] normalvm root[2484]: /etc/xen/scripts/block: Writing backend/vbd/75/51856/physical-device 7:0 to xenstore.
[ 7310.424383] normalvm root[2486]: /etc/xen/scripts/block: Writing backend/vbd/75/51856/hotplug-status connected to xenstore.

So, in a disposable VM, there's three kernel lines. Whereas in the successful case of a non-disposable VM, there's three kernel lines, then the same three lines repeated, then /etc/xen/scripts/block runs.

Or maybe in /var/log/xen/xen-hotplug.log of dispvm?

Doesn't exist, only empty /var/log/xen/xldev.log* files.

Woo! systemctl restart xendriverdomain in the dispvm fixes it!

rustybird · Jun 30, 2016

Woo! systemctl restart xendriverdomain in the dispvm fixes it!

Interesting (but not surprising). Probably should be restarted automatically after DispVM startup (= restore from savefile). Or maybe even ordered to start at that time?

marmarek · Jun 30, 2016

Interesting (but not surprising). Probably should be restarted automatically after DispVM startup (= restore from savefile). Or maybe even ordered to start at that time?

Reordering seems best. Part of QubesOS/qubes-core-agent-linux#18

rustybird · Jul 1, 2016

Reordering seems best. Part of QubesOS/qubes-core-agent-linux#18

Automated announcement from builder-github

The package python2-dnf-plugins-qubes-hooks-3.2.6-1.fc23 has been pushed to the r3.2 testing repository for the Fedora fc23 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

QubesOS/qubes-issues

Join GitHub today

Attaching block devices *from* dispVM broken #2126

Comments

rustybird commented Jun 29, 2016

Qubes OS version (e.g., R3.1):

Affected TemplateVMs (e.g., fedora-23, if applicable):

andrewdavidwong added bug C: core labels Jun 30, 2016

This comment has been minimized.

rustybird commented Jun 30, 2016 • edited Edited 1 time rustybird edited Jun 30, 2016 (most recent)

This comment has been minimized.

rustybird commented Jun 30, 2016 • edited Edited 1 time rustybird edited Jun 30, 2016 (most recent)

This comment has been minimized.

marmarek commented Jun 30, 2016

This comment has been minimized.

rustybird commented Jun 30, 2016

This comment has been minimized.

rustybird commented Jun 30, 2016

This comment has been minimized.

marmarek commented Jun 30, 2016

marmarek added this to the Release 3.2 milestone Jun 30, 2016

This comment has been minimized.

rustybird commented Jul 1, 2016

marmarek self-assigned this Jul 11, 2016

marmarek closed this in marmarek/old-qubes-core-agent-linux@fbf4c93 Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 13, 2016

marmarek added the r3.2-fc23-cur-test label Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 13, 2016

marmarek added the r3.2-fc24-cur-test label Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 13, 2016

marmarek added the r3.2-jessie-cur-test label Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 13, 2016

marmarek added the r3.2-stretch-cur-test label Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 13, 2016

marmarek added the r3.2-wheezy-cur-test label Jul 13, 2016

This comment has been minimized.

marmarek commented Jul 28, 2016

marmarek added r3.2-fc23-stable and removed r3.2-fc23-cur-test labels Jul 28, 2016

This comment has been minimized.

marmarek commented Jul 28, 2016

marmarek added r3.2-fc24-stable and removed r3.2-fc24-cur-test labels Jul 28, 2016

This comment has been minimized.

marmarek commented Aug 31, 2016

marmarek added r3.2-jessie-stable and removed r3.2-jessie-cur-test labels Aug 31, 2016

marmarek added a commit to QubesOS/qubes-core-agent-linux that referenced this issue Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-fc21-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-fc22-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-fc23-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-jessie-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-stretch-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Nov 20, 2016

marmarek added the r3.1-wheezy-cur-test label Nov 20, 2016

This comment has been minimized.

marmarek commented Dec 4, 2016

marmarek added r3.1-fc21-stable and removed r3.1-fc21-cur-test labels Dec 4, 2016

This comment has been minimized.

marmarek commented Dec 4, 2016

marmarek added r3.1-fc22-stable and removed r3.1-fc22-cur-test labels Dec 4, 2016

This comment has been minimized.

marmarek commented Dec 4, 2016

marmarek added r3.1-fc23-stable and removed r3.1-fc23-cur-test labels Dec 4, 2016

This comment has been minimized.

marmarek commented Dec 4, 2016

marmarek added r3.1-jessie-stable and removed r3.1-jessie-cur-test labels Dec 4, 2016

Attaching block devices from dispVM broken #2126

Qubes OS version (e.g., `R3.1`):

Affected TemplateVMs (e.g., `fedora-23`, if applicable):

rustybird commented Jun 30, 2016 •

edited

Edited 1 time

rustybird edited Jun 30, 2016 (most recent)

rustybird commented Jun 30, 2016 •

edited

Edited 1 time

rustybird edited Jun 30, 2016 (most recent)

marmarek closed this in `marmarek/old-qubes-core-agent-linux@fbf4c93` Jul 13, 2016