Nested Virtualization for WSL2 VM

[ameeno]

Greetings,

I am trying to do
Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true

On the wsl2vm itself so I can use KVM inside it.

However, I cannot do that, and I don't know why Get-VM does not see the wsl2vm so I cannot expose these vars.

how do I do it?

[Biswa96]

You can get Hyper-V VM ID with hcsdiag list command as admin. But I don't know if nested virtualization can be done in WSL2. It's like Inception... 🤓

[ameeno]

Hi Biswa96 yes I will try and make it like Inception WIn10 -> wsl(VM) -> KVM -> OSX->Parrallels->Win10 :)

[therealkenc]

I'm on KVM->Win10->WSL2 FWIW.

[kterhorst]

I am interested in this as well. Is there any way to enable ExposeVirtualizationExtensions for the Hyper-V instance running WSL2?

[ameeno]

Well my main desktop rig has been rebuilt around ryzen 3600. Unfortunately, windows does not support nested virtualization in any form for and processors. So it seems a linux Kvm host as the base operating system is required now.

You can enable nested in kvm and have osx and windows 10 SMS with pic box pass through which is almost natively Experiance. However you need two graphics cards. 1 for linux and 1 for your vm.

[scottjoshuac]

This would be very useful to have, that way I don't have to have two separate Linux VMs (WSL2 + my GNS3 VM). Please enable this feature as it's already built into Hyper-V.

[tsujp]

Is there any comment on if and when this will be a thing? It's kind of a requirement for WSL2 to be actually useful, not being able to run VMs with KVM for hardware acceleration in 2019 is a bit.. concerning.

I used WSL1 and it topped out being useful at letting me type Linux commands as it used to have all manner of port issues, restrictions, slowdowns and the like. WSL2 might be faster with I/O but VMs are a CORE REQUIREMENT of any remotely professional setup.

If Microsoft want people to use Windows with WSL2 as a host operating system, WSL2 must support KVM or there needs to be another non-Intel way to get hardware virtualization working.

[steffengy]

Technically there's already everything in place for it to work.

If someone wants to play with this right now with an insider build:

1. Build a kernel with KVM support (missing kvm_intel & msr mostly).
2. Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel.
   This is very hacky.
   Let me know, if you can't wait or are interested in a 1-click workaround.

Et voilà

Point 1) in Microsofts upstream kernel might lead to a bit of discussion, but with both
kvm_intel & kvm_amd as kernel modules everybody should be happy?
For 2) it's either allowing configuration or just passing true.

[scottjoshuac]

Yep, they just need to enable the "ExposeVirtualizationExtensions" feature as it's only available to Hyper-V machines that are in the Hyper-V manager. Since the WSL2 VM does not have a name, we can't apply it.

[steffengy]

@scottjoshuac
Technically you can as the screenshot above shows, it's just not easy at all because
it doesn't seem to be an intended feature yet ;)
(The WSL2 VM is hosted through HCS, therefore doesn't show up in hyperV,
which the powershell CMDLets use. Also a new VM is created each time, so for a workaround
you need to intercept its creation and set ExposeVirtualizationExtensions before its started up)

[rick-pri]

What I'd like to do, and so which requires the WSL 2 isolated VM to be started every time with the ExposeVirtualizationExtensions option, is run the likes of Minikube within WSL 2. Having to have another VM running to then run minikube.exe from within WSL is a real horrible cludge.

Basically, I'm looking for the GUI desktop support of Windows, with the *nix support of WSL in Windows Terminal; as an alternative to MacOS. This has been working okay, until I try to get Minikube to run within the Hyper-V VM that is WSL 2; which it can't of course.

Adding the parameter seems to be a no brainer, it's already there in the technology so there's not really any dev time needed, just testing, although I can see how having a VM running in the background when you kill your terminal session might prove to be a bit of a headache to handle though.

The other point about AMD support is valid too, we need to see this support coming through as well, especially given AMD's current acendency in desktop and laptops.

[rick-pri]

So, I got fed up with this situation today and decided to install Ubuntu in a Hyper-V VM with the ExposeVirtualizationExtensions flag passed to it. I installed the gcloud sdk, installed the components, skaffold, minikube and kubectl and then installed virtualbox-6.1. One slow minikube start later and I can see this:

And so my point from above, turning this on works and so it would be much more helpful if it was enabled for WSL 2.0, either by default or as a configurable, because then I wouldn't have to roll my own VM and add the flag manually. It's just the little additional touches which will make this so much more user focused.

[offlinehacker]

@steffengy how do you intercept creation and set ExposeVirtualizationExtensions, do you maybe have some script available?

[steffengy]

@offlinehacker Sure, it's not very user-friendly though: https://gist.github.com/steffengy/62a0b5baa124830a4b0fe4334ccc2606

[Biswa96]

There are some changes in WSL2 Linux kernel repo with v4.19.104. The config files has some KVM features enabled. So, one may compile the kernel now and try it. No hacky step required.

[steffengy]

@Biswa96 seems like it, though I'd argue that the hacky step is not compiling the kernel but fiddling with HCS.

[Biswa96]

HCS (Hyper-V Compute System) APIs isn't fully** documented for WSL2. But compiling Linux kernel is.

** but it can be found in Windows SDK. And also Docker Desktop uses it.

[steffengy]

@Biswa96 Except that using the HCS API doesn't help here. You currently have to interfere with how LxssManager interacts with HCS and due to LxssManager being a protected process you have to do a lot of hackery (as shown in the GIST) to intercept API calls in HCS internals, which is entirely undocumented besides debug symbols. So yeah building the kernel in anycase is the very very easy part.
If it's implemented in windows eventually and this newer kernel ships, ofcourse you won't have to do anything anymore.

[offlinehacker]

@steffengy Thank you for sharing.

Ok i tried windbg script, but on my build of windows 19041.81, i do not get {"Owner":"WSL"' magic string. I see {"PropertyTypes":["Memory"]} and {"VirtualNodeCount":1,"VirtualMachineMemory":{"AvailableMemory":62,"AvailableMemoryBuffer":832,"ReservedMemory":1048576,"AssignedMemory":1048576,"SlpActive":false,"BalancingEnabled":false,"DmOperationInProgress":false},"VirtualNodes":[{"VirtualNodeIndex":0,"PhysicalNodeNumber":0,"VirtualProcessorCount":8,"MemoryUsageInPages":1048576}]}, but nothing else, even when starting wsl.exe after shutting it down.

Which build of windows are you running?

This method is really hacked up, but i kind of expected that you need to hook dll methods, what else to expect in windows.

[steffengy]

@offlinehacker
Works for me at 19041.84.
Make sure to start a CMD with administrative privileges (else LxssManager is not being restarted) in the folder where you placed the files from the GIST and then run "start-wsl.bat" (double clicking it does not work due to CWD).

[offlinehacker]

@steffengy I have modified your script and managed to get it work. Don't need two step process anymore, directly read, parse, modify and write memory. https://gist.github.com/offlinehacker/4d48f46fdcd64c9a24e3952c69bfcd09

[steffengy]

@offlinehacker Great that you enjoy it, I updated the gist above with your changes. :)

[scottjoshuac]

@steffengy and @offlinehacker thanks for sharing this! I may switch back to Windows now... Although I'm getting used to Linux as my primary and have Wine running the only Windows app I "need" (League of Legends) so it's a tough choice!

[offlinehacker]

I am always using Linux for development, windows was just a gateway drug, as I want to have portable dev environment, that also runs in hyper-v 😅 Just look at ridiculous hacks we need to do, to change a json string passing from one service to other, where in Linux it would be 5 minute fix. Microsoft has done a lot for open source in last years, but as long as windows will not be more open, it is too rigid platform for a lot of developers.

[offlinehacker]

@steffengy thanks for updating gist 🙂

[cyraid]

There's gotta be a better way to keep my wsl2 up to date rather than converting my entire OS (that's already pretty established) to a new insider ring, etc. Can I just download a new package of WSL2 somewhere?

[jseguillon]

I can confirm this works in Win 11. I created a hyperv VM on my Windows 10 with ExposeVirtualizationExtension and was able to launch a nested VM with Kubevirt. A lot of inception but works great: Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊
EDIT: Intel cpu

[crramirez]

I can confirm this works in Win 11. I created a hyperv VM on my Windows 10 with ExposeVirtualizationExtension and was able to launch a nested VM with Kubevirt. A lot of inception but works great: Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊

Intel or AMD ?

[jseguillon]

@crramirez : Intel (edited my comment)

[crramirez]

Hello,

I tested with Windows 11 released version and the latest kernel update (not the WSL Store Preview) and I can confirm that the nested virtualization for AMD is enabled and working by default. The kvm_amd module is included in the kernel and started with nested=1. I tested it using livemedia-creator with qemu and the time to process dropped by half when yesterday I upgraded from Windows 10 to 11.

Regards

[NNTin]

@startergo You didn't eventually compile the kernel?

I have put the same WSL2 config:

[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1

and I get:

cat /sys/module/kvm_intel/parameters/nested
Y

,but at the same time I get this:

dmesg | grep kvm
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.119133] Kernel command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

and

egrep -c '(vmx|svm)' /proc/cpuinfo
0

My CPU supports both VTx and VTd.

I also got this far. For future people if you also got this far try enabling Windows Hypervisor Platform.
https://superuser.com/questions/1510172/hyper-v-vs-virtual-machine-platform-vs-windows-hypervisor-platform-settings-in-p
I had Hyper-V enabled. In BIOS Virtualization Technology an VT-d Feature were enabled but I kept getting

[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

After enabling Windows Hypervisor platform I finally got

[    0.193019] KVM: vmx: using Hyper-V Enlightened VMCS
[    0.195115] kvm: already loaded the other module

[udovenkoav1981]

I have AMD 4800H, windows 11 21h2 (22000.856), wsl kernel 5.10.102.1-microsoft-standard-WSL2
dmesg | grep kvm
[ 0.265398] kvm: no hardware support
[ 0.266298] kvm: Nested Virtualization enabled
[ 0.266306] SVM: kvm: Nested Paging enabled

cat /sys/module/kvm_amd/parameters/nested
1

nested virtualization NOT working, Qemu is very slow

[CMingTseng]

我可以確認這在 Win 11 中有效。我在Windows 10上創建了一個hyperv VM，並且能夠使用Kubevirt啟動一個嵌套VM。很多開始，但效果很好：ExposeVirtualizationExtension``Win10 -> HyperV Win11 -> WSL2 -> Minikube -> Kubvevirt 😊編輯：英特爾中央處理器
Dear Sir

How enable nested virtualization/KVm at Window 10 22H1/22H2 ?

THX

[CMingTseng]

@startergo你最終沒有編譯內核嗎？
我已經放置了相同的 WSL2 配置：

[wsl2]
nestedVirtualization=true
debugConsole=true
pageReporting=true
kernelCommandLine=intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1

我得到：

cat /sys/module/kvm_intel/parameters/nested
Y

，但同時我得到這個：

dmesg | grep kvm
[    0.000000] Command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.119133] Kernel command line: initrd=\initrd.img panic=-1 pty.legacy_count=0 nr_cpus=8 intel_iommu=on iommu=pt kvm.ignore_msrs=1 kvm-intel.nested=1 kvm-intel.ept=1 kvm-intel.emulate_invalid_guest_state=0 kvm-intel.enable_shadow_vmcs=1 kvm-intel.enable_apicv=1
[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

和

egrep -c '(vmx|svm)' /proc/cpuinfo
0

我的 CPU 同時支援 VTx 和 VTd。

我也走到了這一步。對於未來的人們，如果你也走到了這一步，請嘗試啟用Windows虛擬機監控程序平臺。https://superuser.com/questions/1510172/hyper-v-vs-virtual-machine-platform-vs-windows-hypervisor-platform-settings-in-p 我啟用了Hyper-V。在 BIOS 虛擬化技術中啟用了 VT-d 功能，但我不斷得到

[    0.330695] kvm: no hardware support
[    0.330696] kvm: no hardware support

啟用Windows虛擬機管理程序平臺后，我終於得到了

[    0.193019] KVM: vmx: using Hyper-V Enlightened VMCS
[    0.195115] kvm: already loaded the other module

Dear Sir

Do you use Windows 10 22H1/22H2 ?

it can run https://github.com/budtmo/docker-android ?

THX

[startergo]

I have not used it in a while.

[CMingTseng]

@startergo & all

as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config

the ms still disable "nestedVirtualization" at win10

the laste version WSL (download from store)

at Win11 it can run nestedVirtualization

at win10 still not work

PS C:\Users\Lan> wsl --version
WSL 版本： 1.0.3.0
核心版本： 5.15.79.1
WSLg 版本： 1.0.47
MSRDC 版本： 1.2.3575
Direct3D 版本： 1.606.4
DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows版本： 10.0.19045.2486

WinDbg PreView error log

(dd0.17ac): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00007ffa`b5650bb0 cc int 3
0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq
Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq'
onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。
(dd0.1754): Unknown exception - code 800706ba (first chance)
(dd0.1754): Unknown exception - code 800706ba (first chance)
(dd0.1754): Unknown exception - code 800706ba (first chance)

THX

[crramirez]

@CMingTseng, by any chance do you have an AMD processor? Because nested virtualization with AMD is not supported at all in Windows 10

[ZeppLu]

@startergo & all

as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config

the ms still disable "nestedVirtualization" at win10

the laste version WSL (download from store)

at Win11 it can run nestedVirtualization

at win10 still not work

PS C:\Users\Lan> wsl --version WSL 版本： 1.0.3.0 核心版本： 5.15.79.1 WSLg 版本： 1.0.47 MSRDC 版本： 1.2.3575 Direct3D 版本： 1.606.4 DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows版本： 10.0.19045.2486

WinDbg PreView error log

(dd0.17ac): Break instruction exception - code 80000003 (first chance) ntdll!DbgBreakPoint: 00007ffa`b5650bb0 cc int 3 0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq' onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。 (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance)

THX

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

[sarim]

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215

dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2
It takes several minutes to load.

[Simbaclaws]

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215

dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2
It takes several minutes to load.

hey @sarim ,

What kind of hardware is this running on?
Perhaps the cpu is missing specific instruction sets for virtualization acceleration. Since kvm is reporting no hardware support?

That's just from a quick glance through your supplied log.

Hopefully that can be helpful.

Other then that, I currently don't have the time/resources to validate your report. If anyone else can jump in to see wether this is reproducible?

[sarim]

@Simbaclaws Thanks for taking a look. The cpu is AMD Ryzen 7 3700X. It has full support for virtualization.

Ouput of lscpu from wsl.

lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 3700X 8-Core Processor
    CPU family:          23
    Model:               113
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            7200.03
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pd
                         pe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 m
                         ovbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoex
                         t perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1
                          xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmloa
                         d umip rdpid
Virtualization features:
  Virtualization:        AMD-V
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    16 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[Simbaclaws]

@Simbaclaws Thanks for taking a look. The cpu is AMD Ryzen 7 3700X. It has full support for virtualization.

Ouput of lscpu from wsl.

lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 3700X 8-Core Processor
    CPU family:          23
    Model:               113
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            7200.03
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pd
                         pe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 m
                         ovbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoex
                         t perfctr_core ssbd ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1
                          xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmloa
                         d umip rdpid
Virtualization features:
  Virtualization:        AMD-V
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    16 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

@sarim

perhaps some specific flags for acceleration are missing.

you can use this stackoverflow answer to do a quick test what is and is not supported by your cpu:

https://stackoverflow.com/questions/66277749/how-do-i-get-every-supported-cpu-flags-as-qemu-cpu-flags

not sure if that could yield any potential results.

you'll probably want to enable all flags for your cpu model in order to get better performance.

[sarim]

@Simbaclaws I'm already using cpu mode=host-passthrough in libvirt. So it already has all the flags of host cpu. So I don't think it would yield any beneficial result :/

[CMingTseng]

So I'm getting very slow performance using libvirt inside WSL. It is using nested virtualization. But still its slow, especially loading of grub, the loading kernel, loading initrd steps. Do others encountered similar issues?

wsl --version
WSL version: 1.3.17.0
Kernel version: 5.15.90.4-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22621.2215

dmesg | grep kvm
[    0.156436] kvm: no hardware support
[    0.329847] kvm: Nested Virtualization enabled
[    0.329851] SVM: kvm: Nested Paging enabled
[    0.329852] SVM: kvm: Hyper-V enlightened NPT TLB flush enabled
[    0.329853] SVM: kvm: Hyper-V Direct TLB Flush enabled

egrep -c '(vmx|svm)' /proc/cpuinfo
16

sudo virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
Unable to initialize cgroups: internal error: no cgroup backend available
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I'm running a simple debian image: debian-12-genericcloud-amd64.qcow2 It takes several minutes to load.
Dear Sir

Do you run WSL2 at WIn11 ? or WIn 10

THX

[Simbaclaws]

@Simbaclaws I'm already using cpu mode=host-passthrough in libvirt. So it already has all the flags of host cpu. So I don't think it would yield any beneficial result :/

@sarim

you're right, host-passthrough should already do those things.

what about trying to get the cgroup backends available?

Perhaps you need cgroups v2.

Maybe have a look here?
https://stackoverflow.com/questions/73021599/how-to-enable-cgroup-v2-in-wsl2

[CMingTseng]

@startergo & all
as this post https://learn.microsoft.com/zh-tw/windows/wsl/wsl-config
the ms still disable "nestedVirtualization" at win10
the laste version WSL (download from store)
at Win11 it can run nestedVirtualization
at win10 still not work
PS C:\Users\Lan> wsl --version WSL 版本： 1.0.3.0 核心版本： 5.15.79.1 WSLg 版本： 1.0.47 MSRDC 版本： 1.2.3575 Direct3D 版本： 1.606.4 DXCore 版本： 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows版本： 10.0.19045.2486
WinDbg PreView error log
(dd0.17ac): Break instruction exception - code 80000003 (first chance) ntdll!DbgBreakPoint: 00007ffa`b5650bb0 cc int 3 0:006> bp vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq Couldn't resolve error at 'vmcompute!Marshal::JsonParser::JsonParser;g;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.scriptrun C:\Users\Lan\script.js;.detach;qq' onecore\com\combase\dcomrem\call.cxx(4236)\combase.dll!00007FFAB379B3AC: (caller: 00007FFAB379A208) ReturnHr(4) tid(1754) 800706BA RPC 伺服器無法使用。 (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance) (dd0.1754): Unknown exception - code 800706ba (first chance)
THX

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

@ZeppLu

so we can not upgrade WSL2 version ?

THX

[CMingTseng]

2. Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel.
Dear Sir

At WIndows 10 use the WSL2 (kernel version 5.15.79.1 )

How Pass ExposeVirtualizationExtensions to the utility VM and boot a different kernel. ?

THX

[sarim]

@CMingTseng windows 11. you can see my windows version in wsl --version output.

@Simbaclaws I actually already have cgroups v2 enabled. kernel command line to disable cgroupsv1 and mount cgroupsv2. I use it to use podman.

[Simbaclaws]

@CMingTseng windows 11. you can see my windows version in wsl --version output.

@Simbaclaws I actually already have cgroups v2 enabled. kernel command line to disable cgroupsv1 and mount cgroupsv2. I use it to use podman.

@sarim

I'm kind of running out of ideas at this point.

I do see the error message in your log saying it's missing the cgroup backend, but honestly don't know whether fixing it would yield better performance or not.

Hopefully someone else can come up with a solution to your problem.

I can not reproduce this right now since I'm on vacation.

[sarim]

@Simbaclaws Thanks for taking a look. But I suspect It could be bug with WSL, so I was seeking other people's experience with it. If other people's VM's are running fast, then it would be plausible to debug my configuration.

[startergo]

Indeed, the WinDBG hack still works on Windows 10, but only if you don't update wsl.

In short, if wsl --version shows versioning info, then you are bad to go. You should consider downgrading it by:

$pkg = Get-AppxPackage MicrosoftCorporationII.WindowsSubsystemforLinux -AllUsers
Remove-AppxPackage $pkg -AllUsers

Then the WinDBG hack should work as expected!

Correct! Thanks for the input. So, after applying the above version downgrade the WinDbg hack works.