DispVMs no longer start after upgrading dom0 and DVM Template to Xen 4.6.6-37

[andrewdavidwong]

Qubes OS version:

R3.2

Affected component(s):

• DispVMs
• Xen?

---

Steps to reproduce the behavior:

1. Update dom0's Xen packages to 4.6.6-37.
2. Update a Fedora TemplateVM's Xen packages to 4.6.6-37.
3. Make sure that Fedora TemplateVM is the DVM Template.
4. Regenerate the DVM Template.
5. Attempt to start a DispVM.

Expected behavior:

A DispVM is started.

Actual behavior:

No DispVM is started.

General notes:

Also reported on qubes-users:

On 2018-03-17 06:53, qubes-lists@[...] wrote:

DispVM functionality does no longer work for me in Q3.2 since yesterday.

First I assumed the template is broken but after reinstalling the fedora-26 template
from the rpm repo and recreating the DVM I get the very same error.

Has anyone else the same problem after installing the latest dom0 updates?

from the dom0 yum.log the latest installed updates were (2018-03-15):

 libgcc-5.3.1-6.qubes1.fc23.x86_64
 2001:xen-licenses-4.6.6-37.fc23.x86_64
 2001:xen-libs-4.6.6-37.fc23.x86_64
 2001:xen-hypervisor-4.6.6-37.fc23.x86_64
 2001:xen-runtime-4.6.6-37.fc23.x86_64
 libstdc++-5.3.1-6.qubes1.fc23.x86_64
 2001:xen-hvm-4.6.6-37.fc23.x86_64
 2001:xen-4.6.6-37.fc23.x86_64
 qubes-gpg-split-dom0-2.0.28-1.fc23.x86_64
 libgomp-5.3.1-6.qubes1.fc23.x86_64

I also tried a reboot but that did not change anything either.

This is the error I get when trying to start xterm (or anything else) in a DispVM:

echo xterm| /usr/lib/qubes/qfile-daemon-dvm qubes.VMShell dom0 DEFAULT red
time=1521287074.5, qfile-daemon-dvm init
time=1521287074.51, creating DispVM
time=1521287075.04, collection loaded
time=1521287075.05, VM created
time=1521287075.06, VM starting
time=1521287075.06, creating config file
time=1521287075.06, calling restore
Traceback (most recent call last):
  File "/usr/lib/qubes/qfile-daemon-dvm", line 217, in <module>
    main()
  File "/usr/lib/qubes/qfile-daemon-dvm", line 205, in main
    dispvm = qfile.get_dvm()
  File "/usr/lib/qubes/qfile-daemon-dvm", line 167, in get_dvm
    return self.do_get_dvm()
  File "/usr/lib/qubes/qfile-daemon-dvm", line 103, in do_get_dvm
    dispvm.start()
  File "/usr/lib64/python2.7/site-packages/qubes/modules/01QubesDisposableVm.py", line 193, in start
    domain_config, libvirt.VIR_DOMAIN_SAVE_PAUSED)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4405, in restoreFlags
    if ret == -1: raise libvirtError ('virDomainRestoreFlags() failed', conn=self)
libvirt.libvirtError: internal error: libxenlight failed to restore domain 'disp1'

---

Related issues:

[andrewdavidwong]

Note that the Xen package versions match between dom0 and the template. Other problems have been reported due to mismatched versions (e.g., enabling a testing repo in dom0 but not in the template), but this does not appear to be such a case.

[andrewdavidwong]

Note that the Xen package versions match between dom0 and the template. Other problems have been reported due to mismatched versions (e.g., enabling a testing repo in dom0 but not in the template), but this does not appear to be such a case.

[rustybird]

I was constantly affected by something like this this when I was on R3.2. Manually removing the -dvm VM and then running qvm-create-default-dvm worked around whatever that was.

[rustybird]

I was constantly affected by something like this this when I was on R3.2. Manually removing the -dvm VM and then running qvm-create-default-dvm worked around whatever that was.

[andrewdavidwong]

Manually removing the -dvm VM and then running qvm-create-default-dvm worked around whatever that was.

Doesn't work in this case.

[andrewdavidwong]

Manually removing the -dvm VM and then running qvm-create-default-dvm worked around whatever that was.

Doesn't work in this case.

[v6ak]

I have the same or similar issue. Env:

• Qubes 3.2
• dnf list installed | grep ^xen | awk '{print $2}' | sort -u returns a single version: 2001:4.6.6-37.fc23
• happens in both Debian 9 and Fedora 26
• DVM savefile recreation and removal of fedora-26-dvm (or the Debian one) does not help.

It always adds three lines to /var/log/libvirt/libxl/libxl-driver.log:

2018-03-30 20:59:05 CEST xc: error: Restore failed (0 = Success): Internal error
2018-03-30 20:59:05 CEST libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: restoring domain: Success
2018-03-30 20:59:05 CEST libxl: error: libxl_create.c:1147:domcreate_rebuild_done: cannot (re-)build domain: -3

I haven't found anything interesting in other logs, but if think I should check for something, I can.

BTW, I upgraded dom0 before domUs, so I had mismatched versions some time (but TemplateVM upgrade did not help). The issue first appeared some time after boot, because I had some DVMs running, but I was unable to start more of them.

[v6ak]

I have the same or similar issue. Env:

• Qubes 3.2
• dnf list installed | grep ^xen | awk '{print $2}' | sort -u returns a single version: 2001:4.6.6-37.fc23
• happens in both Debian 9 and Fedora 26
• DVM savefile recreation and removal of fedora-26-dvm (or the Debian one) does not help.

It always adds three lines to /var/log/libvirt/libxl/libxl-driver.log:

2018-03-30 20:59:05 CEST xc: error: Restore failed (0 = Success): Internal error
2018-03-30 20:59:05 CEST libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: restoring domain: Success
2018-03-30 20:59:05 CEST libxl: error: libxl_create.c:1147:domcreate_rebuild_done: cannot (re-)build domain: -3

I haven't found anything interesting in other logs, but if think I should check for something, I can.

BTW, I upgraded dom0 before domUs, so I had mismatched versions some time (but TemplateVM upgrade did not help). The issue first appeared some time after boot, because I had some DVMs running, but I was unable to start more of them.

[Polygonbugs]

@andrewdavidwong This also happened to me today. Same error output from starting disp at terminal and /var/log/libvirt/libxl/libxl-driver.log like others.

I'm not sure just upgrading xen package to version 4.6.6-37 invoked the problem. I've already installed all Xen 4.6.6-37 version packages month ago and updated related template several times but soley that didn't make problem. The only thing I've changed is memory policy about VMs so I doubt something related to memory made this problem...

Also removing dvm save file or dvm itself didn't work for me either.

[Polygonbugs]

@andrewdavidwong This also happened to me today. Same error output from starting disp at terminal and /var/log/libvirt/libxl/libxl-driver.log like others.

I'm not sure just upgrading xen package to version 4.6.6-37 invoked the problem. I've already installed all Xen 4.6.6-37 version packages month ago and updated related template several times but soley that didn't make problem. The only thing I've changed is memory policy about VMs so I doubt something related to memory made this problem...

Also removing dvm save file or dvm itself didn't work for me either.

[HW42]

Fix is part of QubesOS/qubes-vmm-xen#36.

[HW42]

Fix is part of QubesOS/qubes-vmm-xen#36.

[qubesos-bot]

Automated announcement from builder-github

The package xen_4.6.6-38+deb9u1 has been pushed to the r3.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen_4.6.6-38+deb9u1 has been pushed to the r3.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package vmm-xen has been pushed to the r3.2 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package vmm-xen has been pushed to the r3.2 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen-4.6.6-38.fc23 has been pushed to the r3.2 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen-4.6.6-38.fc23 has been pushed to the r3.2 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component vmm-xen (including package xen-4.6.6-38.fc26) has been pushed to the r3.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component vmm-xen (including package xen-4.6.6-38.fc26) has been pushed to the r3.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen-4.6.6-40.fc23 has been pushed to the r3.2 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen-4.6.6-40.fc23 has been pushed to the r3.2 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component vmm-xen (including package xen-4.6.6-40.fc26) has been pushed to the r3.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component vmm-xen (including package xen-4.6.6-40.fc26) has been pushed to the r3.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen_2001:4.6.6-40+deb9u1 has been pushed to the r3.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package xen_2001:4.6.6-40+deb9u1 has been pushed to the r3.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update