Windows 10 guest not seeing free memory with balloon driver

[Ristovski]

EDIT: For future reference, virtio memory ballooning is not dynamic by default. Meaning, if you want the guest to request more memory from the hypervisor based on something like memory pressure, that would be up to you to set up (a script talking to a virto channel should work).

Rest of issue is invalid since it's working as currently intended.

---

Enabling the balloon driver works as expected - host memory gets released, but this also causes the Windows 10 guest to think that the released memory is in-use and reports upwards of 99% RAM usage.

Here is a screenshot of enabling and disabling the VirtIO Balloon Driver device in Device Manager.
The memory on the host side does indeed get released and reclaimed as usual, but on the guest side, enabling the driver causes Windows to think that the memory is in-use (thus the peak in the memory graph).

The main issue is that this causes Windows to drop caches to disk, which causes degraded performance, and I assume it also will never get more memory ballooned.

[ybendito]

@Ristovski Please provide more info about the issue: which hypervisor you use: qemu (which?) or something different, how do you inflate the balloon, what is exactly the sequence of operations to reproduce the problem.

[Ristovski]

@ybendito I use virt-manager with qemu-system-x86_64 5.2.0 and the latest version of virtio drivers for Windows.

I don't explicitly balloon it myself, I set the memory settings in virt-manager as follows:

and also have the following entry in the xml:

<memballoon model="virtio" autodeflate="on">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>

From my understanding, the intended behavior would be that the Windows guest would think the memory is free, and the ballooning driver would release/claim more transparently, as its being used.
Instead, the Windows guest idles at 99% RAM and doesn't seem to claim more memory from the host. This means that Windows will not attempt to use any memory caches due to it thinking there is not enough free memory.

I am also able to manually balloon the memory with something like virsh qemu-monitor-command --hmp Win10 'balloon 1500' but again, I don't see how this is intended behavior? Shouldn't the guest automatically request more memory if it has not been explicitly ballooned with a static value?

Whole xml:

Click to expand

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>Win10</name>
  <uuid>6b5473b7-5571-48af-b454-634e9e247df7</uuid>
  <memory unit="KiB">3276800</memory>
  <currentMemory unit="KiB">1048576</currentMemory>
  <vcpu placement="static">2</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-5.2">hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-model" check="partial"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/Win10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1d" function="0x2"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="scsi" index="0" model="virtio-scsi">
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <filesystem type="mount" accessmode="mapped">
      <source dir="/tmp/vmshare"/>
      <target dir="mount"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </filesystem>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <channel type="spiceport">
      <source channel="org.spice-space.webdav.0"/>
      <target type="virtio" name="org.spice-space.webdav.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="2"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <input type="tablet" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </input>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
    </graphics>
    <sound model="ich6">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x01" function="0x0"/>
    </sound>
    <audio id="1" type="spice"/>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio" autodeflate="on">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-netdev"/>
    <qemu:arg value="user,id=net0"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="virtio-net-pci,netdev=net0"/>
  </qemu:commandline>
</domain>

[Ristovski]

For reference, this is how it looks like right after the guest boots up:

Using the guest like this as-is gives terrible performance, since the cache will quickly fill up and get paged to disk (notice the amount of memory available, and the amount cached)

From my understanding, the guest balloon driver should be able to automatically request more memory from the host, but this is not the case (or is happening at 100% RAM usage, in which case its completely useless due to the aforementioned issue with caches and paging)

[ybendito]

IMO, completely correct behavior of the balloon. The guest is configured to run with [max of] 3.2G memory and use 1G. So the rest of memory is in the inflated balloon. When you disable the balloon driver the memory becomes free. When the balloon is enabled you can change the current allocation to be = max, this will deflate the balloon.
@hammerg Do I miss something?

[Ristovski]

@ybendito I assumed the ballooning was dynamic by default, but now I see this is not the case.

Closing as invalid.

[linuzer]

I have the very same observation on that behaviour. And although I have not much insight on the technical details behind the scenes, I have a hard time to understand how this could be expected behaviour...

Rather I would expect the same behaviour than with Hyper-V dynamic memory: From the host you tell the guest the range of memory it is allowed to use. The guest starts with the minimum and leaves the rest to the host. Later on, when the guest starts using more and more memory (e.g. opening some programs), in the same way the guest's used memory increases, the host's free memory decreases. And when you over provisioned the guest's memory, the host would even start swapping out the memory to the hard disk.

I thought the memory balloon tries to implement just this. Am I missing out some conceptual details?

[Ristovski]

@linuzer From what I was able to gather, I assume Windows does not support memory hot add/remove while Linux does. And indeed virtio memory ballooning with a Linux guest works as intended.

Meanwhile with Windows guests the following happens:
Lets say you set the max memory to 4GB, and default to 2GB

• the guest boots and sees 4GB of physical memory attached
• the virtio balloon driver inside Windows fake "reserves"/locks the spare 2GB (I assume this just marks the pages as in-use)
• to Windows, this looks like a driver allocating 2GB of RAM which it can not operate on, while in reality this memory gets released to the host

Now if you were to update the virtio ballooning parameters at runtime, this would simply cause the virtio balloon driver to free or reserve additional memory.

I am not sure how Hyper-V does it (perhaps Windows detects its running under Hyper-V and enables different memory management?), but I assume this is the only alternative.

[linuzer]

OH, I see. Thanks a lot for the explanation! So in other words, the balloon-driver does it the linux-way, which works perfect in a linux guest, but on a Windows guest it just provokes the above results. Of course, changing the parameters on-the-fly could somehow improve the situation.

Would it be right to asume, that the Windows-part of the balloon-driver just needs to be adopted to the Hyper-V-way/Windows-way of memory management to get the same behaviour under Windows?

[vrozenfe]

Doing it the same way as Hyper-V does can be a problem.
Hyper-V dynamic memory (dmvsc.sys) DmcHotAddPageRange function built around MmAddPhysicalMemory DDI which is reserved for system use https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/_kernel/

[ohault]

@vrozenfe In the while, MmAddPhysicalMemory has been documented https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/nf-ntddk-wdm-mmaddphysicalmemory

[vrozenfe]

@ohault
Thanks a lot for the information.
AFAIK Microsoft did it quite recently, still claiming that the following functions MmAddPhysicalMemoryEx, MmRemovePhysicalMemory and MmRemovePhysicalMemoryEx are " reserved for system use" https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/_kernel/

Best,
Vadim.

[qixinwuchen]

@Ristovski windows vm only use 1 GB ram，and I allocate it for 3 GB ram， why rss display 3 GB？
the vm has already install virtio balloon driver, and configured memballoon tag in libvirt xml，
I think rss should display 1 GB，right？

3707e1622a1147965e90781c446bfa16 3707e162-2a11-4796-5e90-781c446bfa16 data_verify_2.216 8377344 8377344 15 KEPTDATA keptdata_verify One2Data hvm destroy restart destroy /usr/libexec/qemu-kvm