VMs don't boot after qubes-dom0-update

[ohreally]

Qubes OS version:

R4.0rc3

Affected TemplateVMs:

None (dom0 issue).

---

Steps to reproduce the behavior:

qubes-dom0-update

Expected behavior:

Software gets updated. Stuff works as before, but better, because updated.

Actual behavior:

After a reboot, none of the VMs that are set to start when the system boots, are started. Except for dom0, fortunately.

The relevant line in systemctl status qubes-vm@sys-net.service is

dom0 qvm-start[PID]: Not enough memory to start domain 'sys-net'

(Same for the other VMs.)

I can, however, manually start sys-usb, sys-net, sys-firewall and sys-whonix, and then everything seems to run as usual. I even have enough memory left to start several other VMs.

General notes:

I ran qubes-dom0-update with no arguments.
I ran the update on the 10th of January, and I rebooted on the 14th.
To be sure it was not a temporary glitch, I rebooted again (twice), but the problem persists.

At the same time as the dom0 update, I also updated both the fedora-25 and debian-8 templates, but that does not seem to be related to the problems I encounter.

May be related

qvm-ls.txt shows the output of qvm-ls -O NAME,STATE,MEMORY.
Looks strange to me, but I never looked at this output before, so maybe it's always been like this.

[marmarek]

Can you post logs from qmemman service from about that time?

[marmarek]

Can you post logs from qmemman service from about that time?

[ohreally]

qmemman.log.txt shows qmemman log from reboot + a few minutes (first line before reboot to include PID change).
This laptop has 8GB of memory.

[ohreally]

qmemman.log.txt shows qmemman log from reboot + a few minutes (first line before reboot to include PID change).
This laptop has 8GB of memory.

[marmarek]

What version of qubes-core-dom0 do you have? Similar problem was fixed in 4.0.18.

[marmarek]

What version of qubes-core-dom0 do you have? Similar problem was fixed in 4.0.18.

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.21-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.21-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.21-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.21-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[ohreally]

So, I just updated dom0 (and fedora-26 template). And now sys-net and sys-usb boot fine on startup, but the others don't. When I try and boot any of the other virtual machines (CLI or GUI, no difference), I get a message saying

libxenlight failed to create new domain 'personal', see /var/log/libvirt/libxl/libxl-driver.log for details

And that logfile says

2018-02-06 22:51:53.693+0000: xc: panic: xc_dom_elfloader.c:66: xc_dom_guest_type: image not capable of booting inside a HVM container: Invalid kernel
2018-02-06 22:51:53.694+0000: libxl: libxl_dom.c:677:libxl__build_dom: xc_dom_parse_image failed: Invalid argument
2018-02-06 22:51:53.714+0000: libxl: libxl_create.c:1284:domcreate_rebuild_done: cannot (re-)build domain: -3

Funny thing is, that if I open the Qube settings for the VM in question, and go to tab 'Advanced', I see it is in PVH mode. And if I set it to HVM mode, it boots fine.

[ohreally]

So, I just updated dom0 (and fedora-26 template). And now sys-net and sys-usb boot fine on startup, but the others don't. When I try and boot any of the other virtual machines (CLI or GUI, no difference), I get a message saying

libxenlight failed to create new domain 'personal', see /var/log/libvirt/libxl/libxl-driver.log for details

And that logfile says

2018-02-06 22:51:53.693+0000: xc: panic: xc_dom_elfloader.c:66: xc_dom_guest_type: image not capable of booting inside a HVM container: Invalid kernel
2018-02-06 22:51:53.694+0000: libxl: libxl_dom.c:677:libxl__build_dom: xc_dom_parse_image failed: Invalid argument
2018-02-06 22:51:53.714+0000: libxl: libxl_create.c:1284:domcreate_rebuild_done: cannot (re-)build domain: -3

Funny thing is, that if I open the Qube settings for the VM in question, and go to tab 'Advanced', I see it is in PVH mode. And if I set it to HVM mode, it boots fine.

[marmarek]

PVH require kernel >= 4.11. Make sure it is set there (default should be 4.14.13-3).

[marmarek]

PVH require kernel >= 4.11. Make sure it is set there (default should be 4.14.13-3).

[ohreally]

The only available option on my system is 4.9.56-21.

[ohreally]

The only available option on my system is 4.9.56-21.

[marmarek]

Try installing updates, there should be newer kernel-qubes-vm package.

[marmarek]

Try installing updates, there should be newer kernel-qubes-vm package.

[ohreally]

I guess something got corrupted here, then:

# qubes-dom0-update --refresh
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
Last metadata expiration check: 0:00:00 ago on Wed Feb 7 07:19:29 2018.
Dependencies resolved.
Nothing to do.
Complete!
No packages downloaded

# dnf info kernel-qubes-vm
Last metadata expiration check: 0:03:55 ago on Wed Feb 7 07:20:31 2018.
Installed Packages
Name : kernel-qubes-vm
Arch : x86_64
Epoch : 1000
Version : 4.9.56
Release : 21.pvops.qubes
Size : 208 M
Repo : @System
From repo : anaconda
Summary : The Xen Kernel
URL : http://www.kernel.org/
License : GPL v2 only
Description : Qubes domU kernel.

I'll dive deeper into this, next week or so; I have very little time this week.

[ohreally]

I guess something got corrupted here, then:

# qubes-dom0-update --refresh
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
Last metadata expiration check: 0:00:00 ago on Wed Feb 7 07:19:29 2018.
Dependencies resolved.
Nothing to do.
Complete!
No packages downloaded

# dnf info kernel-qubes-vm
Last metadata expiration check: 0:03:55 ago on Wed Feb 7 07:20:31 2018.
Installed Packages
Name : kernel-qubes-vm
Arch : x86_64
Epoch : 1000
Version : 4.9.56
Release : 21.pvops.qubes
Size : 208 M
Repo : @System
From repo : anaconda
Summary : The Xen Kernel
URL : http://www.kernel.org/
License : GPL v2 only
Description : Qubes domU kernel.

I'll dive deeper into this, next week or so; I have very little time this week.