Support Qemu accelerated podman machine on Windows hosts

[arixmkii]

/kind feature

Description

Support Qemu accelerated podman machine on Windows hosts.

• gvproxy supports windows in its latest releases https://github.com/containers/gvisor-tap-vsock/releases/tag/v0.3.0
• There are Qemu windows builds, which supports Hyper-V based acceleration https://qemu.weilnetz.de/w64/ so, -machine q35,accel=whpx:tcg should perform snappy on Hyper-V enabled hosts.

I'm pretty new to podman machine, so, I might be not aware of other external components needed to run podman machine.

I saw that there is support for running podman machine in WSL2 being merged to 4.0.0 RC2. Why this idea still makes sense.

• It will be Fedora CoreOS based;
• It will be fully featured Linux VM;
• It will more precisely match behavior of the podman machine on MacOS;
• It will allow running at least some arm64 images with qemu syscalls for guest architecture on windows hosts.

The downside from WSL2 option would be less efficient host resource management, but in some cases it is not of the concern.

What is missing right now is the command line support for choosing between multiple available vendors for podman machine on a single platform, which I don't think is a thing already (at least from my studying of the source tree).

[rhatdan]

I am fine with this, but we need community to work on it. If you want to implement this and it is simply a CLI change, I would be all for it.

[afbjorklund]

@arixmkii

The main problem was the lack of unix sockets, for the Windows build. Then there were a lot of other "little things", to fix.

Here some old code, if you want to continue with it, https://github.com/afbjorklund/podman/commits/machine-windows

Things like missing "xzcat.exe" and missing $HOME, and so on and so forth.

The official Podman support would be for WSL2, so it (Windows) was abandoned.

• Introduce Windows WSL implementation of podman machine #12503

Like you say, qemu itself should support windows and co-exist with Hyper-V.

[arixmkii]

@afbjorklund Thank you for your input! I have interest to at least try to make it work, but I need to study the code in more details, it will take time.

The official Podman support would be for WSL2

Feel free to close this issue then with this resolution, it could be reopened/recreated later if there is some actual work to base it on.

[afbjorklund]

There was windows support in podman-machine (v1) but it mostly used VirtualBox so the QEMU driver was bare-bones...

It should be possible to use "fifo files" instead of "unix sockets", but it will take some adaption of the go code and qemu flags.

[albertdb]

This would be really nice to have. Currently the only alternative, where WSL2 is not feasible, is using Minikube, which at the current version includes Podman v2.

[afbjorklund]

You could also DIY, using something like Vagrant for the provisioning. But otherwise I think "everyone" is using WSL2 now.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/supported-guest-os

[arixmkii]

The main problem was the lack of unix sockets

Go has built in support for unix sockets on windows https://go-review.googlesource.com/c/go/+/125456/ which I expect could work just fine for gvproxy and podman binaries.

Mingw added afunix.h last summer (pretty fresh and probably will need to build it from sources to get it into toolchain) https://github.com/mingw-w64/mingw-w64/blob/43e87a27fdb97ae562f7c1e2017c8ce58fef9ee1/mingw-w64-headers/include/afunix.h I believe then Qemu sources could be patched to allow unix sockets in windows build at least behind conditional switch.

Windows support lacks DATAGRAM for unix sockets, but I'm not ready to say if this will be a blocker or not.

missing "xzcat.exe"

Download latest windows build of xzutils https://tukaani.org/xz/ and copy or remane xz.exe to xzcat.exe and it will behave as xzcat, because it checks the command line first argument to choose the behavior. More natural with symlinks, but this way it will work on windows - checked SHA256 of processed qcow2.xz on Win and MacOS hosts.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[arixmkii]

There is a patch submitted to QEMU for enabling builds with af_unix support on Windows https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg00221.html

[arixmkii]

This also looks useful https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg04098.html as the functionality of file descriptors for unix sockets on windows seems to be limited in Go lang/runtime (probably platform limitations).

It will be needed to replace gvproxy qemu_wrapper pattern from podman.

[arixmkii]

@rhatdan I started making additional changes - to allow building installer packages with QEMU support. This is needed before creating PR, so, that interested ones would have straight forward way to check how it works. Hope to come up with a draft PR this weekend, where I could discuss with podman team the bits of changed code, which are not perfectly fit with the current codebase.

[arixmkii]

@rhatdan I created a draft PR for this #16872 Marked as draft, because I expect that there will be some amount of comments and questions from development team. Especially about utilizing environment variables to select the machine provider.

[arixmkii]

The very first user friendly release (packaged into installers and more compact) https://github.com/arixmkii/qcw/releases/tag/v0.0.7

It should not be considered production ready, but can be used for testing or evaluation purposes.

P.S. QEMU installer (built with 9pfs on Windows) is now being built in the CI and will be added later.

P.P.S. Installers are unsigned of course, so, they might not work on some machines, where policies for only signed software are enforced.

[arixmkii]

Worked with 9p Windows support for QEMU developers. We traced down 3 bugs in the current implementation. There is a hotfix ready for tests. Will try to publish build with FS mount support soon.

[arixmkii]

Example of QEMU Pomdan machine with p9 on Windows

Creating the machine

podman machine init --image-path testing --username core --cpus 4 --memory 8192 -v C:\Temp\PodmanStorage;/home/core/ps;rw -v C:\Temp\PodmanReadonly;/home/core/pr;ro

Here I modified command line argument parsers to use os.PathListSeparator (';' on Windows platform) to simplify usage for demo purposes, I will create a separate issue to work on the parser, which will be acceptable for the mainlining. 2 directories mounted. One RW to /home/core/ps and another RO to /home/core/pr (now I understand that this naming is bad for the demo, but let it stay)

Then start machine as normal (I did with debug logs just in case, but this should not matter)

Then launching interactive terminal inside container with both mounts

podman run -it --rm -v /home/core/ps:/mnt/write -v /home/core/pr:/mnt/read busybox

Work with readonly FS

### Using RO FS

~ # cd /mnt/read/
/mnt/read # cat ReadMe.txt
RO text
/mnt/read # echo "added" > ReadMe.txt
sh: can't create ReadMe.txt: Read-only file system
/mnt/read # echo "added" > Other.txt
sh: can't create Other.txt: Read-only file system

And the with read-write FS

### Using RW FS

/mnt/read # cd /mnt/write/
/mnt/write # cat ModifyMe.txt
sample
/mnt/write # echo "new sample" >> ModifyMe.txt
/mnt/write # cat ModifyMe.txt
sample
new sample
### This is actually a bug
/mnt/write # echo "only sample" > ModifyMe.txt
sh: can't create ModifyMe.txt: Invalid argument
/mnt/write # echo "other sample" > Other.txt
/mnt/write # cat Other.txt
other sample
/mnt/write # rm Other.txt
/mnt/write # cat Other.txt
cat: can't open 'Other.txt': No such file or directory
/mnt/write # mkdir inner
/mnt/write # ls -l
total 0
-rw-rw-rw-    1 nobody   nobody          18 Jan 12 20:55 ModifyMe.txt
drwxrwxrwx    1 nobody   nobody           0 Jan 12 20:59 inner
/mnt/write # ls -l inner
total 0

There discovered bug with the permissions will be reported to people working on QEMU side. There are more known issues with the current implementation:

• content of directories with symlinks (Windows symlinks) can't be enumerated
• content of directories with AF_UNIX sockets (Windows Unix domain sockets) can't be enumerated
• file enumeration is not safe, when content of the directory is modified during enumeration
• can't overwrite the file (this one was discovered during this demo)

Summary would, that in general it works almost like on macOS. Ok, one can't really use -v "$HOME:$HOME" for the machine and then magical -v "$PWD:$PWD" for the container, when working directory is under user home, but this was always like "hiding the actual complexity" thing, not a real feature (though looking like the one).

Will prepare and publish demo builds (both QEMU and Podman) in the coming days.

[arixmkii]

Finally managed to publish pre-built packages https://github.com/arixmkii/qcw/releases/tag/v0.0.8 and updated README file to reflect recent changes.

[arixmkii]

I would say that now we have all the bricks ready and just need to put them together.

• LookPath like impelementations for OS specific executable searches common#1131 - and then need to wait for the new common version to be pulled into Podman tree
• Support for Windows paths in the source position of the volume mounts #17113 - nice to have, but 9pfs implementation will be only be included in future version of QEMU, for now this change will only work if one builds QEMU with patches
• Implement Unix domain socket support for VLAN #17473 - main and most significant part, review pending
• [Feature]: Support Podman Machine provider selection using containers.conf #17116 - will require one more PR, when new common version is pulled into the tree. Based on this commit arixmkii@ce9e930

More in the pipeline:

• Consider adding machine provider prefix to ssh keys #16710 - nice to have usability fix, but is not a blocker.

[arixmkii]

Published pre-built packages including all PRs mentioned above https://github.com/arixmkii/qcw/releases/tag/v0.0.9 There are hopes, that the next time Podman package will be built from the unmodified source tree 🤞

[arixmkii]

Published latest QEMU and Podman builds under https://github.com/arixmkii/qcw/releases/tag/v0.0.11

All included changes applied to Podman are now also submitted as PRs and are waiting for being reviewed:

• Implement Unix domain socket support for VLAN #17473
• Use dynamic identity file names by default #18487
• Enable QEMU Podman machine on Windows #18488

Unfortunately 9pfs on Windows didn't make it to QEMU 8.0.0, so, the patched QEMU build (provided from the release I mentioned) is still needed to experiment with it.

[arixmkii]

Some noteworthy updates:

• managed to make it run nested (with SMP enabled and 4 CPUs) inside Hyper-V Win 11 machine (only tested with Insider Preview for now)

[arixmkii]

I published new builds based on Podman 4.7.0-dev and QEMU 8.1.0 with support for running inside Hyper-V machine https://github.com/arixmkii/qcw/releases/tag/v0.0.12

[arixmkii]

Another incremental release with patches rebased (and ready for review) to latest Podman 4.8.0-dev and QEMU 8.1.1
https://github.com/arixmkii/qcw/releases/tag/v0.0.13

[arixmkii]

New build, now code changes are validated with the newly introduced cross platform e2e Podman Machine test suite.

All future builds will be validated with that test suite. Going forward I will just update the link in this comment to point to latest rebuild if there are no significant functional changes.

https://github.com/arixmkii/qcw/releases/tag/v0.0.16

Updated Jan, 06, 2024: v0.0.16 released moving to QEMU 8.2.0 and Podman 5.0.0-dev

[jgresham]

@arixmkii First, thanks for your work on this! It would be nice for me to remove the WSL2 install flow to use podman in my app on Windows.

How is the stability & performance of Podman on QEMU + Windows looking? It seems like most users home laptops don't have HyperV, so will that be a problem here?

[arixmkii]

@jgresham I can confirm that it works on Win 11 Home with "Virtual Machine Platform" feature enabled and there is no requirement of the full Hyper-V feature installed.

As for performance, I can't comment on this, because after recent struggles I moved all my development into VMs on my Windows laptop, so, I'm running Podman machine inside Hyper-V VM on a portable HW and it is not as snappy as bare metal for sure. You can share some ideas of benchmarks you would like to see and I may be able to run them on my desktop machine, but no promises of timeline here.

As for stability. I use it for assisting me doing my dev work on Windows, which is relatively lightweight scenario, as I'm on Windows only for my personal projects. I try to fix issues, when I encounter them, but I admit I had never stress tested the QEMU machine. I'm running full set of Podman machine e2e tests on the commit I place my published releases on, to at least prevent obvious regressions. I'm also using the same QEMU changes related in Podman app on my MacOS setup just to have it slightly more tested.

[arixmkii]

Everything is moved to the new Podman Machine 5 refactoring, this required rework to multiple parts. The first preview build is provided here (QEMU is still building and will be added in couple of hours): https://github.com/arixmkii/qcw/releases/tag/v0.0.17 I plan to do another build after Podman 5 is released, because the source tree around the machine code is receiving multiple changes and fixes now.

There are still some hopes, that this change #21594 could be accepted to Podman Machine 5, which will dramatically reduce the amount of additional code needed to support Windows.

[hgkamath]

@jgresham :

It seems like most users home laptops don't have HyperV, so will that be a problem here?

just penning for FYI: ANS: Most likely, No, it won't be a problem. For Windows Home edition, there are batch-files that can be found online that augment the feature set of the Windows Home Edition. The batch files just download and install the components that provide the respective feature, which are normally only bundled with the Professional-Edition. The package lists are determined from%SystemRoot%\servicing\Packages\*.mum files, and packages are downloaded by the Windows tool DISM from official sources. The WinOS becomes a configuration unsupported by Microsoft, but it works. The WHPX hypervisor can be used with qemu and virtualbox. WSL also works. WRT the HyperV manager and HyperV-Manager-Manged-Machines, I haven't fully pushed them to the max, but for small experiments that I did with it some years ago, the type-1 and type-2 hyperV-VM-s worked. On Win11-Home, the WSA (which as of this writing is being sunset) also works.

• gpedit-enabler.bat Group Policy Editor
• Hyper-V-Enabler.bat HyperV and HyperV manager

So 'Windows Home' users who want to use WHPX virtualization just have this extra step. One, could also argue that 'Windows Home' users elected to use a non-virtualization OS, but one could also argue that virtualization based software is not only for business and enterprise users, but necessary for home users too. The processor/motherboard has virtualization capabilities that the WinOS Home-edition disallows to use. Microsoft should provide more to its home edition users for a more functional home-use laptop.