No Qubes/VMs starting - libxenlight failed to create new-domain in 4.2.1

[scallyob]

WARNING: This issues is only above very old AMD CPUs, not supported by AMD anymore. Some workaround listed below have severe security consequences, do not apply them unless you really understand all the implications!

How to file a helpful issue

Qubes OS release

4.2.1

Brief summary

I did a full update on April 23 and rebooted April 24. First reboot since April 1.
Now no Qubes/VMs will start.

Steps to reproduce

1. Update dom0 and all VMs.
2. reboot

Expected behavior

VMs set to autostart start up

Actual behavior

"libxenlight failed to create new-domain" pop ups for sys-net, sys-firewall, etc

qvm-ls - shows all VMs halted

/var/log/libvirt/libxl/libxl-driver.log shows:

libxl: libxl_dm.c:2857:stubdom_xswait_cb: Domain 1: Stubdom 2 for 1 startup: startup timed out
libxl: libxl_create.c:1975: domcreate_devmodel_started: Domain 1:device model did not start -9

This repeats many times with the Domain # changing

(there are also errors related to PCI device, but these are present for previous boots as well and are not new. The errors above do not appear until today in the log.)

[krystian-hebel]

Good thing I refreshed list of issues, I was about to report the same. I can confirm that VMs were restarted right after upgrade and they worked until following boot.

I also checked that after removing all network controllers from sys-net and changing its type to PVH all VMs can be started (obviously without networking), so this seems to be a problem with HVM.

This happened on KGPE-D16 with Opteron 6282 SE, with ASUS firmware 3001 (i.e. no coreboot).

[scallyob]

KGPE-D16 with Opteron 6282 SE

Oh no, I'm on same hardware!

[scallyob]

Kernel downgrade in dom0 and sys-net did not produce different results.

[marmarek]

What can you see in /var/log/xen/console/guest-sys-net-dm.log ?

[scallyob]

There's a lot in there, not sure what to look for. Don't see errors or warnings.
(talked about this here: https://forum.qubes-os.org/t/no-qubes-vms-boot-after-latest-updates/26033/14)

[marmarek]

Errors can be buried quite deep there... look for anything after starting qemu.

[scallyob]

I see it start qemu, with a bunch of options, each time it tries to boot. Nothing obvious to me after that that seems like a problem.

[krystian-hebel]

What can you see in /var/log/xen/console/guest-sys-net-dm.log ?

For me it ends with:

If full log may contain something useful I may try to get it tomorrow, without network it will require some finesse.

[marmarek]

Indeed nothing obvious there... But one worrying thing is the timing: the "Rescanning PCI Frontend" messages are on stubdomain cleanup, and based on timestamps it's pretty close to starting qemu. AFAIR the startup timeout is 10s, but usually the stubdomain startup takes below 1s. This is pretty old system, it may be that recent workaround for speculative-execution bugs made it significantly slower.

Is it with current-testing (in dom0) enabled or not? Best to identify which update specifically broke it. dnf history may help, but my guess is Xen package. There is also Xen update in current-testing since yesterday, maybe this one will help?

[krystian-hebel]

Is it with current-testing (in dom0) enabled or not?

No, just default ones.

I also think this may be caused by Xen, that would explain why initial restart of VMs succeeded and only after full reboot they won't come up, will check different versions later.

[scallyob]

Is it with current-testing (in dom0) enabled or not?

No.

Tried downgrading to: xen-hvm-stubdom-linux-4.2.9-1.fc37.x86_64.rpm AND xen-hvm-stubdom-linux-full-4.2.9-1.fc37.x86_64.rpm

No change.

Tried downgrading xen, xen-hypervisor, xen-libs, xen-licenses and xen runtime to 4.17.3-4, but it gave me error:

The operation would result in removing the following protected packages: qubes-core-dom0

So not sure how to proceed.

[marmarek]

Tried downgrading xen, xen-hypervisor, xen-libs, xen-licenses and xen runtime to 4.17.3-4, but it gave me error:

The operation would result in removing the following protected packages: qubes-core-dom0

So not sure how to proceed.

xen package needs to match exact version of python3-xen - so you need this one too

[scallyob]

Ok, that allowed the downgrade, now everything is booting.

CORRECTION: everything seems to boot EXCEPT for sys-usb

start failed: Timed out during operation: cannot acquire state change lock (held by monitor=shutdown-event-20) [Fixed: caused by switching from HVM to PV]