Known QEMU2 Driver Issues

[sharifelgamal]

Now that #13639 is (soon to be) merged, we need a place to aggregate all the known issues so we can eventually graduate it from experimental to recommended driver.

• containerd runtime is broken (permissions on the socket file)
• crio runtime is broken (podman and crio-wipe return errors trying to mount a volume)
• minikube service (implemented with Implement socket_vmnet network (QEMU) #14989)
• minikube tunnel (implemented with Implement socket_vmnet network (QEMU) #14989)
• Full test suite on macOS arm64
• Full test suite on linux arm64
• Unconfigurable firmware paths
• Registry addon (kube-registry-proxy images has no arm64 version) (implemented with addons registry: Use multi-arch kube-registry-proxy image #16577)
• Windows support (find the correct qemu-system command line args) (experimental support implemented with Implement experimental QEMU on Windows code #15781)
• Cannot specify custom api server port
• Use VDE/Socket drivers for macOS networking (implemented with Implement socket_vmnet network (QEMU) #14989)
• Multi-node (based on Add multi-node support for QEMU #14527)

[afbjorklund]

Networking currently for "user" (slirp) only. Proper networking on Mac is possible, by using the VDE and Socket drivers from Lima. But not implemented in driver yet - should be easy to do, though

• https://github.com/lima-vm/vde_vmnet
• https://github.com/lima-vm/socket_vmnet

The default network should still be supported (through ssh tunneling, like docker driver) but when using these networks the machine will get an IP and direct access from the host is possible.

https://wiki.qemu.org/Documentation/Networking

[afbjorklund]

The current tunneling of the docker port is kinda gross, but it was blocked on upstream not allowing any changes...

• Engine port 2376 should not be hardcoded docker-archive-public/docker.machine#3361

i.e. the docker port was hardcoded to 2376, so currently it does some kind of bait-and-switch on the URL instead

Now when the driver is forked, should remove that workaround - and add the tunneling of the minikube 8443 port.

#13934 (comment)

[afbjorklund]

Naming: once the dust settles, the "qemu2" driver should be aliased as the "qemu" driver - similar to how the "kvm2" driver eventually overtook the original "kvm" driver. The forks are more an historical fact, and machine upstream is now dead.

• https://github.com/machine-drivers/docker-machine-driver-qemu

• https://github.com/machine-drivers/docker-machine-kvm (libvirt)

Note: the "qemu" and "kvm" drivers are for docker-machine, while the "qemu2" and "kvm2" drivers are for minikube
The original drivers still work (with patches), but the project is not officially supported by Docker Inc. anymore (since 2019).

[afbjorklund]

The QEMU2 driver is supposed to work on all three platforms, with hardware acceleration, but it needs more testing.
On Linux it uses KVM, on macOS it uses HVF and on Windows it uses WHPX. The qemu parameter is -accel.

anders@ubuntu:~$ uname
Linux
anders@ubuntu:~$ arch
x86_64
anders@ubuntu:~$ qemu-system-x86_64 -accel help
Accelerators supported in QEMU binary:
tcg
kvm
anders@ubuntu:~$ qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
tcg

The above output means that amd64 is accelerated, but that arm64 is emulated (through "TCG", similar to Rosetta).
It is possible to run the virtual machine in emulation, but it is not supported and it is much (10x) slower and has timeouts.

https://wiki.qemu.org/Documentation/TCG

[afbjorklund]

@sharifelgamal : issue with workaround for lots of memory: #14273

Should be possible to remove the workaround, on recent macOS