Incus & LXD Windows 11 VMs work on Ubuntu 22.04 with limits.memory=16GiB - But FAIL on Ubuntu 24.04

[bmullan]

HOST system

12 core
64GB ram
btrfs file systems

---

Boot off an Ubuntu 22.04 SSD

Ubuntu 22.04

• LXD version 5.21.1
• incus version 6.1
• Kernel 6.5.0.35

Create a new LXD Windows 11 VM using a backup LXD Windows11 VM image tarball

Create a new Incus Windows 11 VM using the Same backup LXD Windows11 VM image tarball

Set Both LXD and Incus VMs config.memory to 8GiB

$ lxc config set lxdwin11 limits.memory=8GiB
$ incus config set incuswin11 limits.memory=8GiB

Start both the Incus and LXD Windows 11 VMs

BOTH work as expected !

Set Both LXD and Incus VMs config.memory to 16GiB

$ lxc config set lxdwin11 limits.memory=16GiB
$ incus config set incuswin11 limits.memory=16GiB

Start both the Incus and LXD Windows 11 VMs

BOTH work as expected !

---

Using same HOST system.

Boot off an Ubuntu 24.04 SSD

Ubuntu 24.04

• LXD version 5.21.1
• incus version 6.0.0
• Kernel 6.8.0.0-31

Create a new LXD Windows 11 VM again using the same backup LXD Windows11 VM image tarball

Create a new Incus Windows 11 VM again using the same backup LXD Windows11 VM image tarball

Set Both LXD and Incus VMs config.memory to 8GiB

$ lxc config set lxdwin11 limits.memory=8GiB
$ incus config set incuswin11 limits.memory=8GiB

Start both the Incus and LXD Windows 11 VMs

BOTH work as expected !

Set Both LXD and Incus VMs config.memory to 16GiB

$ lxc config set lxdwin11 limits.memory=16GiB
$ incus config set incuswin11 limits.memory=16GiB

This is where FAILURE occurs with both LXD and Incus Windows 11 VMs

Start both the Incus and LXD Windows 11 VMs

**Both LXD and Incus Windows 11 VMs with 16GB memory:

• Start to boot
• Get an IPv6 but no IPv4
• After 10-15 seconds BOTH VMs exhibit max CPU Utilization
• then BOTH VMs Terminate and enter the "Stopped" State"**

Something with Ubuntu 24.04 has changed that prevents configuring an LXD or Incus Windows 11 VM
with limits.memory=16GiB while it works on Ubuntu 22.04.

[stgraber]

Look for incus info --show-log NAME in case QEMU decided to print something useful.
Also try with raw.qemu=-cpu host to see if that somehow helps (we've seen cases where it does for Windows guests).

[bmullan]

I'll try that today.

Odd thing to me is on 24.04 limits.memory is the only config option that has a problem.

For example, limits.cpu use works w any valid # of cpu.

[bmullan]

Initial Log for New copy of Incus Windows 11 VM

$ incus info --show-log win11-test
Name: win11-test
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/05/21 07:35 EDT
Last Used: 1969/12/31 19:00 EST
Error: open /var/log/incus/win11-test/qemu.log: no such file or directory

==========================================================================

$ incus config set win11-test limits.memory=16GB

$ incus start win11-test

==========================================================================

Once win11-base starts, it gets an IPv4

$ incus ls
+------------+---------+--------------------+------------------------------------------------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------------+---------+--------------------+------------------------------------------------+-----------------+-----------+
| incuswin11 | STOPPED | | | VIRTUAL-MACHINE | 0 |
+------------+---------+--------------------+------------------------------------------------+-----------------+-----------+
| win11-base | STOPPED | | | VIRTUAL-MACHINE | 0 |
+------------+---------+--------------------+------------------------------------------------+-----------------+-----------+
| win11-test | RUNNING | 10.65.2.191 (eth0) | fd42:650e:5ff8:ee26:f8ec:2368:448c:8fbe (eth0) | VIRTUAL-MACHINE | 0 |
| | | | fd42:650e:5ff8:ee26:216:3eff:fe95:3fce (eth0) | | |
+------------+---------+--------------------+------------------------------------------------+-----------------+-----------+

==========================================================================

20-30 seconds later...

"Watching" $ incus ls win11
+------------+---------+------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------------+---------+------+------+-----------------+-----------+
| incuswin11 | STOPPED | | | VIRTUAL-MACHINE | 0 |
+------------+---------+------+------+-----------------+-----------+
| win11-base | STOPPED | | | VIRTUAL-MACHINE | 0 |
+------------+---------+------+------+-----------------+-----------+
| win11-test | STOPPED | | | VIRTUAL-MACHINE | 0 |
+------------+---------+------+------+-----------------+-----------+

==========================================================================

$ incus info --show-log win11-test

$ incus info --show-log win11-test
Name: win11-test
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/05/21 07:35 EDT
Last Used: 2024/05/21 07:39 EDT

Log:

Log is Empty ?

=======================================================

$ next I set win11-test to include raw.qemu -cpu host

$ incus config show win11-test
architecture: x86_64
config:
limits.memory: 16GB
raw.qemu: -cpu host
volatile.base_image: ac9a07c35632b1a32ee2584abc4b40132212ffd6a8e04e9d67a0ff288f20e404
volatile.cloud-init.instance-id: e61021eb-2a68-473e-9dd1-b665b7984a0a
volatile.eth0.hwaddr: 00:16:3e:95:3f:ce
volatile.last_state.power: STOPPED
volatile.last_state.ready: "false"
volatile.uuid: 2ba302a4-8ad8-4801-86dd-9de6bcf97590
volatile.uuid.generation: 2ba302a4-8ad8-4801-86dd-9de6bcf97590
volatile.vsock_id: "1279105970"
devices: {}
ephemeral: false
profiles:

• default
  stateful: false
  description: ""

==============================================================

Next, retarted win11-test VM

After 60-90 seconds the Host itself freezes and has to be reset with the Power Button!

NOTE

BEFORE setting raw.qemu -cpu host the same Windows 11 VM would experience High CPU
but for a much shorter Period of time and then the Windows 11 VM would just enter the "Stopped" state (ie the Host did not freeze). But with raw.qemu -cpu host the Host itself freezes & its Pwr Button time

Also, strange to me is that after either with or without ** raw.qemu -cpu host** the win11-test VM
could - Start, High CPU then Terminate/Stop or Start VM & Host Freezes requiring Pwr Button.

And NO Log entries get written during all those events/time (see below)?

$ incus info --show-log win11-test

Name: win11-test
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/05/21 07:35 EDT
Last Used: 2024/05/21 08:32 EDT

Log:

[stgraber]

The fact that it messes things up to the point of needing a hardware reboot makes it sound like a potential kernel issue.

Any chance you can try the kernel from https://github.com/zabbly/linux?

That would get you a "clean" (no distro patches) Linux kernel, currently a 6.8.10.

[bmullan]

Stephane

I installed the Zabbly kernel & rebooted.

First tried just 4GB memory

$ incus config set win11 limits.memory=4GB

and the Windows 11 VM boots and gets an IPv6, then IPv4
VM ... works as expected

*$ incus config set win11 limits.memory=8GB
same as w/4GB memory limit - works as expected

$ incus config set win11 limits.memory=16GB

System starts, VM gets IPv6 & IPv4 then after
about 30-45 seconds there is HIgh CPU then the Windows 11 VM
terminates (stops)

Next...

$ incus config set win11 raw.qemu -- "-cpu host -device intel-hda -device hda-duplex -audio spice"

Start the incus Windows 11 VM with "-cpu host" and "limits.memory=16GB"

$ incus ls
shows win11 gets IPv6 & IPv4

BUT... again (and this is with the Zabbly Kernel)
after 60-90 seconds the HOST itself freezes requiring use of Power Button to reset it.

Its got to be something that is common the both Incus & LXD.
like... kernel, version of qemu, the BTRFS file system?

If kernel caused

• its something that works in Ubuntu 22.04 kernel
  that does not in 24.04 kernel
  If qemu caused
• 22.04 and 24.04 default Qemu versions are different
• BTRFS ? just throwin that one in

So summary so far...

Using the same TAR file to create the new Windows 11 VMs on both Ubuntu 22.04 and Ubuntu 24.04

On Ubuntu 22.04 LTS
vm limits.memory=16GB .... works

On Ubuntu 24.04
vm limits.memory=16GB .... VM starts, gets ipv4/6 then high cpu then terminates -- But Host is alive

On Ubuntu 24.04 (default 24.04 kernel)
vm limits.memory=16GB
and
raw.qemu cpu host

.... VM starts, gets ipv4/6 then high cpu then the Host itself freezes requiring Power Button.

On Ubuntu 24.04 (Zabbly Kernel)

Exact same results as with non-Zabbly Kernel.

[bmullan]

@stgraber

Wanted to give you an update on this issue with limits.memory

Not sure what got fixed as I did 2 things on my Ubuntu 24.04 this morning:

• $ sudo apt update && sudo apt upgrade -y
• Changed from Incus 6.0 LTS to Incus Stable

Regardless, the problem with VM limits.memory=8GB is fixed somewhere.

I'm running a Windows11 VM with limits.cpu=2 limits.memory=*GB

And this system is not crashing/freezing any longer.