Creating a VM with numa fails - Parameter -numa node,mem is not supported

[hprem]

Issue

With numa_nodes specified in the Vagrantfile, vagrant up fails with the error

qemu-system-x86_64: -numa node,nodeid=0,cpus=0-1,mem=4096: Parameter -numa node,mem is not supported by this machine type
Use -numa node,memdev instead

Steps to reproduce

Just do vagrant up with the below-mentioned Vagrantfile

Expected behaviour

VM should come up

Actual behaviour

vagrant up fails

System configuration

Debian

Libvirt version:

❯ dpkg -l | grep libvirt0
ii  libvirt0:amd64                          5.0.0-4+deb10u1                         amd64        library for interfacing with different virtualization systems

Output of vagrant version; vagrant plugin list:

❯ vagrant version; vagrant plugin list
Installed Version: 2.2.16
Latest Version: 2.2.16

You're running an up-to-date version of Vagrant!
vagrant-libvirt (0.5.3, global)
  - Version Constraint: > 0

Attach Output of VAGRANT_LOG=debug vagrant ... --provider=libvirt >vagrant.log 2>&1

 INFO interface: error: There was an error talking to Libvirt. The error message is shown
below:

Call to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2021-06-17T15:30:08.200134Z qemu-system-x86_64: -numa node,nodeid=0,cpus=0-1,mem=4096: Parameter -numa node,mem is not supported by this machine type
Use -numa node,memdev instead
There was an error talking to Libvirt. The error message is shown
below:

Call to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2021-06-17T15:30:08.200134Z qemu-system-x86_64: -numa node,nodeid=0,cpus=0-1,mem=4096: Parameter -numa node,mem is not supported by this machine type
Use -numa node,memdev instead
 INFO interface: Machine: error-exit ["VagrantPlugins::ProviderLibvirt::Errors::FogError", "There was an error talking to Libvirt. The error message is shown\nbelow:\n\nCall to virDomainCreateWithFlags failed: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead\n2021-06-17T15:30:08.200134Z qemu-system-x86_64: -numa node,nodeid=0,cpus=0-1,mem=4096: Parameter -numa node,mem is not supported by this machine type\nUse -numa node,memdev instead"]

A Vagrantfile to reproduce the issue:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/bullseye64"

  # Provider-specific configuration
  config.vm.provider "libvirt" do |libvirt|
     libvirt.cpus = 4
     libvirt.numa_nodes = [
      {:cpus => "0-1"},
      {:cpus => "2-3"}
     ]
  end
end

Are you using upstream vagrant package or your distros package?
Upstream

[electrofelix]

This looks like a libvirt <--> qemu mis-match where the domain XML is correct as far as libvirt is concerned but it's generating the wrong arguments when it in turn runs qemu.

Can you grab the domain XML generated using virsh dumpxml <name> and share.

[hprem]

Here is the generated domain xml config file. Should we report this to the libvirt team?

❯ sudo virsh dumpxml test_default
<domain type='kvm'>
  <name>test_default</name>
  <uuid>8ecfeee6-9d38-42a2-a51b-4287441d5a55</uuid>
  <description>Source: /home/hprem/vagrant/test/Vagrantfile</description>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <numa>
      <cell id='0' cpus='0-1' memory='0' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='0' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test_default.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='52:54:00:a4:5d:0b'/>
      <source network='vagrant-libvirt'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

[electrofelix]

While there is no harm confirming my guess on what is happening, I suspect it will ultimately need to be reported to the distro rather than libvirt. I can't be completely sure, but I'd guess that the distro has updated qemu past the point of what is fully supported by the version of libvirt provided by the distro packages. For most this has no impact, just in this case making use of the one part that changed in qemu that is not backwards compatible.

As I don't use this feature myself I can't be completely sure that the following is correct, which is why it might be worth asking about this on the libvirt mailing list.

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <numa>
      <cell id='0' cpus='0-1' memory='0' unit='KiB'/>
      <cell id='1' cpus='2-3' memory='0' unit='KiB'/>
    </numa>
  </cpu>

One check we could look to try is see if we can boot a VM using an image matching your distro version when it was initially released and then see how booting a nested works, separately can check this against a few other recent distro releases and those should help in reporting to the distro assuming it confirms the issue.

[electrofelix]

Running the provided Vagrantfile on ubuntu 20.04 works perfectly fine:

ii  libvirt0:amd64                                6.0.0-0ubuntu8.9                           amd64        library for interfacing with different virtualization systems

qemu:

dpkg -l qemu
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version           Architecture Description
+++-==============-=================-============-======================================
ii  qemu           1:4.2-3ubuntu6.16 amd64        fast processor emulator, dummy package

Will spin up a buster image and install vagrant and then attempt to launch a nested VM to see if I can reproduce.

[hprem]

I am using a recent version of qemu, installed from the official buster-backports repository

❯ dpkg -l | grep qemu-system-x86
ii  qemu-system-x86                         1:5.2+dfsg-9~bpo10+1                    amd64        QEMU full system emulation binaries (x86)

[electrofelix]

@hprem that's the problem I'd say, you need a more recent version of libvirt as well.

I can successfully stand up the VM on vanilla debian 10:

vagrant@buster:~$ vagrant up --provider libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'debian/buster64' version '10.20210409.1' is up to date...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 192.168.123.88:22
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Creating shared folders metadata...
==> default: Installing NFS client...
==> default: Exporting NFS shared folders...
==> default: Preparing to edit /etc/exports. Administrator privileges will be required...
==> default: Mounting NFS shared folders...

==> default: Machine 'default' has a post `vagrant up` message. This is a message
==> default: from the creator of the Vagrantfile, and not from Vagrant itself:
==> default: 
==> default: Vanilla Debian box. See https://app.vagrantup.com/debian for help and bug reports
vagrant@buster:~$ cat /etc/issue
Debian GNU/Linux 10 \n \l

With packages:

dpkg -l qemu libvirt0
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version              Architecture Description
+++-==============-====================-============-=============================================================
ii  libvirt0:amd64 5.0.0-4+deb10u1      amd64        library for interfacing with different virtualization systems
ii  qemu           1:3.1+dfsg-8+deb10u8 amd64        fast processor emulator, dummy package

I'd guess that qemu 5.x has changed some of the command line options compared to what is used for numa nodes.

If you can't backport a more recent version of libvirt, you can look at dropping the numa section:

     libvirt.numa_nodes = [
      {:cpus => "0-1"},
      {:cpus => "2-3"}
     ]

And specify instead a qemuargs section:

     libvirt.qemu_args ....

That will allow you to pass through the corresponding args directly if you know the right combination. It's not ideal, but should allow you to work around the limitations of your current version of libvirt. It may not be sufficient because I don't know too much about what else might be needed for the memdev attribute to work as I'm assuming there is some additional device that must be defined for this.

[electrofelix]

The other solution would be to boot a newer debian 11 as a VM with libvirt.nested = true enabled, and then within that VM boot your desired config (assuming the version of qemu is new enough in bullseye).

[hprem]

@electrofelix
Thanks for looking into this issue. I guess I would go up with libvirt.qemu_args .... for now.
There isn't a backported version of libvirt available for buster. I will follow up with the debian maintainers to check why the there is some mismatch with the libvirt version available in buster and backported version of qemu

[hprem]

And to confirm, everything works perfectly fine with debian11. But I am stuck with using the stable version for now