Enable Nested Virtualization on Mac OS 15

[aparashk]

Continuation of #3405

Mac's Hypervizor.Framework supports nested virtualization starting with Mac OS 15.0 (Sequoia), if the host CPU is M3 or later.
We need this to be also supported in UTM, so that UTM Linux VMs can make use of KVM.

[andmpel]

https://developer.apple.com/documentation/virtualization/vzgenericplatformconfiguration/4360552-isnestedvirtualizationenabled?changes=latest_minor

Relevant apple documentation

[andriysavin]

So what about nested virtualization for Windows guest? Low level hypervisor API seems to support it as well https://developer.apple.com/documentation/hypervisor/4359587-hv_vm_config_set_el2_enabled

[osy]

For QEMU, we typically wait for the feature to be implemented upstream or we pick a patch. I think enabling nested virt requires doing more than just setting that flag.

[neverpanic]

This doesn't seem to work for me. I'm on 15.1 (24B83), M3 Pro, UTM 4.6.0 (102), with a freshly created Fedora 40 VM using Hypervisor.framework, and it doesn't seem to enable KVM support on boot:

root@localhost:~# dmesg | grep 'kvm'
[    0.050577] kvm [1]: HYP mode not available
root@localhost:~# dmesg | grep 'CPU:'
[    0.026095] CPU: All CPU(s) started at EL1
root@localhost:~# dmesg | grep 'CPU features:'
[    0.000000] CPU features: detected: Address authentication (IMP DEF algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v4
[    0.026107] CPU features: detected: ARMv8.4 Translation Table Level
[    0.026107] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.026108] CPU features: detected: Common not Private translations
[    0.026108] CPU features: detected: CRC32 instructions
[    0.026108] CPU features: detected: Data cache clean to Point of Deep Persistence
[    0.026109] CPU features: detected: Data cache clean to Point of Persistence
[    0.026109] CPU features: detected: Data independent timing control (DIT)
[    0.026109] CPU features: detected: E0PD
[    0.026110] CPU features: detected: Enhanced Privileged Access Never
[    0.026110] CPU features: detected: Generic authentication (IMP DEF algorithm)
[    0.026111] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.026111] CPU features: detected: LSE atomic instructions
[    0.026111] CPU features: detected: Privileged Access Never
[    0.026112] CPU features: detected: RAS Extension Support
[    0.026112] CPU features: detected: Speculation barrier (SB)
[    0.026112] CPU features: detected: TLB range maintenance instructions

Maybe my expectations are wrong, but I would have expected the CPU to be started in EL2 and HYP mode to be available. Am I doing things wrong? Is there a debug log I could check to make sure nested virtualization is actually enabled on this VM?

[neverpanic]

Nevermind, works just fine with Fedora 41, so I'm guessing the issue is on the VM client side. For reference, this is what it looks like when it works as expected:

root@localhost:~# dmesg | grep 'kvm'
[    0.105119] kvm [1]: nv: 529 coarse grained trap handlers
[    0.105154] kvm [1]: IPA Size Limit: 40 bits
[    0.105671] kvm [1]: GICv3: no GICV resource entry
[    0.105674] kvm [1]: disabling GICv2 emulation
[    0.105697] kvm [1]: GIC system register CPU interface enabled
[    0.105705] kvm [1]: vgic interrupt IRQ9
[    0.105753] kvm [1]: Hyp nVHE mode initialized successfully
root@localhost:~# dmesg | grep 'CPU:'
[    0.042219] CPU: All CPU(s) started at EL2

[chodorenko]

At the same not work
MacOS 15.1 Sequoia (24B83)
MacBook Air M3
UTM 4.6.0 (102)
Clean install Fedora Server 41 aarch64 from Fedora-Server-netinst-aarch64-41-1.4.iso
Maybe i do not correct understand howe to enable nested virtualization ?
config file config.plist.zip

[sidkshatriya]

@neverpanic how did you get this to work ? You simply tried with Fedora 41 instead of Fedora 40 ? Or did you make some other configuration changes in UTM ?

[sidkshatriya]

@neverpanic My guess is this worked because you checked the checkbox "Use Apple Virtualization" when creating a new virtual machine ? (This way we don't need to worry about qemu's support for nested virtualization)

[neverpanic]

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

[sidkshatriya]

However, this also didn't work reliably for me.

Thanks for your reply -- Very mysterious indeed. Not able to get this to work at all. Not sure what caused your VM to work suddenly. Maybe only P cores support nested virtualisation ? Maybe addition of certain devices / displays causes nested virtualisation to get disabled ? I tried with various configuration variations of Fedora Server 41 without any luck !

[chodorenko]

Yes, this only works with Apple Virtualization, as documented in the release notes.

However, this also didn't work reliably for me. I had to create multiple VMs, and it worked on some of them. Once it started working on a VM, it generally kept working on that particular VM, though — I just ended up creating multiple, testing whether nested virtualization worked, and then overwriting the disk with the one I wanted to be attached.

You can give your config.plist from VM which Nested Virtualization work ? Or maybe can you find diff which my config from post #6700 (comment) ?