Add the qemu2 driver to the minikube registry #13639

afbjorklund · 2022-02-20T09:38:35Z

Since the machine drivers are hardcoded in minikube,
drivers need to be added to the registry to be tested.

Add a basic sanity check for the qemu-system binary,
and set up the basic configuration such as cpus/memory.

Most likely needs an updated version of the upstream driver...

https://github.com/machine-drivers/docker-machine-driver-qemu

Note: this driver uses qemu and runs in userspace but can use kvm

There is another driver called kvm2, which uses the libvirt library

mprimeaux · 2022-03-05T22:46:08Z

@afbjorklund How can I help to progress this PR?

afbjorklund · 2022-03-06T07:41:21Z

@mprimeaux : first of all, thanks for volunteering!

This particular PR is more about adding "qemu" to the hard-coded list of allowed machine drivers for minikube
Most likely we will end up having to fork a "qemu2" for minikube, like there was a "kvm2" done for libvirt earlier ?

There are two issues going on here, at the same time...

First we need an ISO that we can test with, and second we need an (Open Source) VM driver that can run it.
Since the current ones are behind in their development, we are trying first to update them for amd64 - then arm64.

Ideally, we would want a solution (for aarch64) that could work for minikube as well as for docker-machine.

But that is secondary, working for minikube is primary... Docker stopped all development on the project in 2019.

Our current buildroot is based on BIOS and isolinux, and needs to be changed to use UEFI and grub to boot up.

The current drivers are based on VirtualBox or on HyperKit, which are only available for the x86_64 architecture.

klaases · 2022-03-14T22:20:05Z

Hi @mprimeaux, were you able to review @afbjorklund's comments?

mprimeaux · 2022-03-20T16:48:59Z

@afbjorklund @klaases Thanks for the overall context and related details.

Tactically, I am unsure where to begin as I am not as familiar with the Minikube code base (yet). Seems focusing on a buildroot that uses UEFI and grub is a logical first foray. Thoughts?

afbjorklund · 2022-03-20T18:00:34Z

It is pretty much two separate projects. And it is possible to start with x86_64, and move on to aarch64.

You will find that this PR is mostly about the minikube driver "registry", and doesn't really have any code.

The main reason why it is so big, is that the config part has been forked from libmachine (for some reason)

https://github.com/machine-drivers/docker-machine-driver-qemu/blob/master/qemu.go#L247_L256

The docker-machine drivers use the driver GetCreateFlags and SetConfigFromFlags methods instead.

That means that the drivers can be decoupled (using gRPC), and doesn't have to be included/linked...

So most likely we will end up having to fork "qemu" into a "qemu2", and then adopt it to minikube ?

Note that the driver doesn't really work at the moment, it barely compiles and only for amd64 at that.

afbjorklund · 2022-04-02T08:22:13Z

The driver actually works now, if you upgrade docker-machine and boot2docker and compile it from source...

--qemu-boot2docker-url=https://github.com/afbjorklund/boot2docker/releases/download/3be318c/boot2docker.iso

afbjorklund · 2022-04-02T08:29:09Z

I have added flags to the driver, so that it will be possible to boot different architectures (beyond amd64).

There is a truckload of them, but they can be defaulted. Both machine and cpu are mandatory for arm64.

qemu-program qemu-system-x86_64
qemu-machine (new, optional)
qemu-cpu (new, optional)
qemu-bios (new, bool)
qemu-uefi (new, bool)
qemu-firmware (new) /usr/share/OVMF/OVMF_CODE.fd

Since the machine drivers are hardcoded in minikube, drivers need to be added to the registry to be tested. Add a basic sanity check for the qemu-system binary, and set up the basic configuration such as cpus/memory.

Copy some of the default values from the regular setup, that is not used when not using docker-machine config.

Only the SSH and the Docker port (2376) are available with the slirp network, everything else needs tunneling.

It is still ancient, so there is bound to be issues with the vendored code from libmachine and drivers.

This is an internal driver, with the new driver config rather than the machine flags of the external driver.

Assume "user" networking, for now at least.

afbjorklund · 2022-04-04T18:55:41Z

Now this has both the QEMU (docker-machine) and QEMU2 (new minikube) drivers.

Briefly tested on Linux, needs some more parameters before it works on Darwin too.

Uses SSH for apiserver (8443) tunneling

Basic support for ARM64 parameters...

This reverts commit e7495ab.

…emu2

sharifelgamal · 2022-05-11T22:36:42Z

/ok-to-test

minikube-pr-bot · 2022-05-11T23:38:58Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13639) |
+----------------+----------+---------------------+
| minikube start | 53.9s    | 54.5s               |
| enable ingress | 30.7s    | 31.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 30.2s 30.8s 30.7s 31.2s 30.7s
Times for minikube (PR 13639) ingress: 30.2s 31.1s 31.2s 30.7s 32.1s

Times for minikube start: 52.6s 52.9s 55.0s 54.2s 54.6s
Times for minikube (PR 13639) start: 55.3s 54.0s 54.3s 54.3s 54.6s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13639) |
+----------------+----------+---------------------+
| minikube start | 24.3s    | 23.7s               |
| enable ingress | 22.7s    | 22.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 22.9s 22.4s 22.4s 22.9s 22.9s
Times for minikube (PR 13639) ingress: 21.9s 21.9s 23.4s 23.5s 22.9s

Times for minikube start: 26.0s 23.4s 24.9s 23.9s 23.3s
Times for minikube (PR 13639) start: 23.8s 23.2s 23.3s 24.0s 23.9s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13639) |
+----------------+----------+---------------------+
| minikube start | 32.4s    | 35.6s               |
| enable ingress | 24.6s    | 21.6s               |
+----------------+----------+---------------------+

Times for minikube start: 29.7s 28.2s 28.1s 32.0s 44.0s
Times for minikube (PR 13639) start: 33.0s 39.6s 44.3s 28.3s 32.8s

Times for minikube ingress: 22.4s 22.4s 32.4s 17.9s 27.9s
Times for minikube (PR 13639) ingress: 17.9s 21.9s 27.4s 21.9s 18.9s

minikube-pr-bot · 2022-05-12T01:46:05Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux_containerd	TestNetworkPlugins/group/cilium/Start (gopogh)	6.17 (chart)
Docker_Windows	TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh)	11.11 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/auto/DNS (gopogh)	16.05 (chart)
Docker_Linux_containerd	TestAddons/parallel/Registry (gopogh)	17.44 (chart)
Docker_Linux	TestNetworkPlugins/group/auto/DNS (gopogh)	20.20 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/custom-weave/Start (gopogh)	23.46 (chart)
Docker_Linux_containerd	TestOffline (gopogh)	27.43 (chart)
Docker_Linux_containerd	TestPause/serial/Start (gopogh)	27.43 (chart)
Docker_Windows	TestPause/serial/Pause (gopogh)	29.41 (chart)
Docker_Linux_containerd	TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh)	34.15 (chart)
Docker_Linux	TestFunctional/serial/ComponentHealth (gopogh)	36.00 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MountCmd/any-port (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MySQL (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/PersistentVolumeClaim (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmd (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmdConnect (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/serial/ExtraConfig (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/serial/SoftStart (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/serial/StartWithProxy (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestMultiNode/serial/DeployApp2Nodes (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	37.17 (chart)
Docker_Linux_containerd	TestFunctional/parallel/DashboardCmd (gopogh)	38.05 (chart)
Docker_Windows	TestNetworkPlugins/group/enable-default-cni/DNS (gopogh)	45.65 (chart)
Hyper-V_Windows	TestNoKubernetes/serial/StartWithK8s (gopogh)	46.74 (chart)
Docker_macOS	TestNetworkPlugins/group/kubenet/Start (gopogh)	48.15 (chart)
Docker_Windows	TestRunningBinaryUpgrade (gopogh)	53.33 (chart)
Docker_Windows	TestNetworkPlugins/group/kubenet/Start (gopogh)	54.29 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/kindnet/DNS (gopogh)	57.89 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

k8s-ci-robot · 2022-05-12T03:41:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [afbjorklund,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afbjorklund requested review from medyagh and klaases February 20, 2022 09:38

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 20, 2022

k8s-ci-robot requested a review from blueelvis February 20, 2022 09:39

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 20, 2022

afbjorklund force-pushed the driver-qemu branch 4 times, most recently from 45acbec to d00b8b8 Compare February 21, 2022 06:08

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 31, 2022

afbjorklund force-pushed the driver-qemu branch from 065de9b to 359d65c Compare April 2, 2022 08:21

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 2, 2022

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 4, 2022

afbjorklund changed the title ~~Add the qemu driver to the minikube registry~~ WIP: Add the qemu driver to the minikube registry Apr 4, 2022

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 4, 2022

afbjorklund added 6 commits April 4, 2022 20:15

Add the qemu driver to the minikube registry

284ad62

Since the machine drivers are hardcoded in minikube, drivers need to be added to the registry to be tested. Add a basic sanity check for the qemu-system binary, and set up the basic configuration such as cpus/memory.

Upgrade to latest docker-machine-driver-qemu

98b8b6a

Copy some of the default values from the regular setup, that is not used when not using docker-machine config.

Use portforwarding for qemu with user network

d9c303e

Only the SSH and the Docker port (2376) are available with the slirp network, everything else needs tunneling.

Upgrade to latest docker-machine-driver-qemu

927d7d5

It is still ancient, so there is bound to be issues with the vendored code from libmachine and drivers.

Fork the machine qemu driver into minikube qemu2

77ed383

This is an internal driver, with the new driver config rather than the machine flags of the external driver.

Set up the IP address for the qemu driver

6965307

Assume "user" networking, for now at least.

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 4, 2022

afbjorklund added 3 commits April 4, 2022 21:01

Add missing machine type for QEMU2 for test

e894c88

Make sure to set up apiserver tunnel on restart

e7495ab

Make sure to look for KIC before using oci port

d21342c

afbjorklund changed the title ~~Add the qemu driver to the minikube registry~~ WIP: Add the qemu driver to the minikube registry Apr 10, 2022

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 10, 2022

afbjorklund mentioned this pull request Apr 10, 2022

New QEMU2 driver, minikube fork of the QEMU machine driver #13934

Closed

afbjorklund linked an issue Apr 10, 2022 that may be closed by this pull request

New QEMU2 driver, minikube fork of the QEMU machine driver #13934

Closed

afbjorklund added 2 commits April 10, 2022 17:54

Revert "Make sure to set up apiserver tunnel on restart"

15c71df

This reverts commit e7495ab.

Add parameters needed for vde networking

ae40c57

sharifelgamal mentioned this pull request Apr 11, 2022

Build ISO minikube image for ARM (aarch64) #9228

Closed

sharifelgamal added 2 commits May 3, 2022 14:01

fix firmware locations for darwin

3555b19

Merge branch 'master' of github.com:kubernetes/minikube into anders-q…

5b39a46

…emu2

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2022

sharifelgamal added 2 commits May 11, 2022 09:06

Merge branch 'master' of github.com:kubernetes/minikube into anders-q…

00d31d0

…emu2

support darwin hardware acceleration

9a0f1a0

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2022

fix lint

60328d4

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 11, 2022

sharifelgamal changed the title ~~WIP: Add the qemu driver to the minikube registry~~ Add the qemu2 driver to the minikube registry May 11, 2022

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 11, 2022

sharifelgamal mentioned this pull request May 11, 2022

Known QEMU2 Driver Issues #14146

Open

12 tasks

sharifelgamal approved these changes May 12, 2022

View reviewed changes

sharifelgamal merged commit b26dcf2 into kubernetes:master May 12, 2022

klaases mentioned this pull request Jul 26, 2022

WIP: create tunnel for qemu #14615

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the qemu2 driver to the minikube registry #13639

Add the qemu2 driver to the minikube registry #13639

afbjorklund commented Feb 20, 2022 •

edited

mprimeaux commented Mar 5, 2022

afbjorklund commented Mar 6, 2022 •

edited

klaases commented Mar 14, 2022

mprimeaux commented Mar 20, 2022

afbjorklund commented Mar 20, 2022 •

edited

afbjorklund commented Apr 2, 2022 •

edited

afbjorklund commented Apr 2, 2022 •

edited

afbjorklund commented Apr 4, 2022 •

edited

sharifelgamal commented May 11, 2022

minikube-pr-bot commented May 11, 2022

minikube-pr-bot commented May 12, 2022

k8s-ci-robot commented May 12, 2022

Add the qemu2 driver to the minikube registry #13639

Add the qemu2 driver to the minikube registry #13639

Conversation

afbjorklund commented Feb 20, 2022 • edited

mprimeaux commented Mar 5, 2022

afbjorklund commented Mar 6, 2022 • edited

klaases commented Mar 14, 2022

mprimeaux commented Mar 20, 2022

afbjorklund commented Mar 20, 2022 • edited

afbjorklund commented Apr 2, 2022 • edited

afbjorklund commented Apr 2, 2022 • edited

afbjorklund commented Apr 4, 2022 • edited

sharifelgamal commented May 11, 2022

minikube-pr-bot commented May 11, 2022

minikube-pr-bot commented May 12, 2022

k8s-ci-robot commented May 12, 2022

afbjorklund commented Feb 20, 2022 •

edited

afbjorklund commented Mar 6, 2022 •

edited

afbjorklund commented Mar 20, 2022 •

edited

afbjorklund commented Apr 2, 2022 •

edited

afbjorklund commented Apr 2, 2022 •

edited

afbjorklund commented Apr 4, 2022 •

edited