Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kvm2 driver: Use scsi cdrom for arm64 to fix Linux on Apple Silicone #18239

Merged
merged 1 commit into from Mar 8, 2024

Conversation

nirs
Copy link
Contributor

@nirs nirs commented Feb 23, 2024

On linux/aarch64 (e.g. Asahi Linux on MacBook M*) booting from SATA cdrom is broken and the VM drops into the UEFI shell.

It seems that linux/aarch64 supports only virtio and scsi devices[1]. Replace with scsi cdrom (like the x86 version) and addd a virtio-scsi controller since the default scsi controller does not boot as well.

[1] https://kubevirt.io/user-guide/virtual_machines/virtual_machines_on_Arm64/#disks-and-volumes

Fixes #18238

@k8s-ci-robot k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 23, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 23, 2024
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

On linux/aarch64 (e.g. Asahi Linux on MacBook M*) booting from SATA
cdrom is broken and the VM drops into the UEFI shell.

It seems that linux/aarch64 supports only virtio and scsi devices[1].
Replace with scsi cdrom (like the x86 version) and addd a virtio-scsi
controller since the default scsi controller does not boot as well.

[1] https://kubevirt.io/user-guide/virtual_machines/virtual_machines_on_Arm64/#disks-and-volumes
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Feb 23, 2024
@nirs nirs changed the title kvm2 driver: Use scsi cdrom for compabillity with linux/aarch64 kvm2 driver: Use scsi cdrom for arm64 Feb 23, 2024
@nirs
Copy link
Contributor Author

nirs commented Feb 23, 2024

Build and install

make
prog="out/docker-machine-driver-kvm2"
rm -f "$prog"*
make "$prog"
cp "$prog" ~/.minikube/bin/

Starting a cluster

$ out/minikube start --driver kvm2 -p dr1 --network default
😄  [dr1] minikube v1.32.0 on Fedora 39 (arm64)
✨  Using the kvm2 driver based on user configuration
💿  Downloading VM boot image ...
    > minikube-v1.32.1-1708020063...:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > minikube-v1.32.1-1708020063...:  344.04 MiB / 344.04 MiB  100.00% 23.81 M
👍  Starting control plane node dr1 in cluster dr1
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.28.4 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "dr1" cluster and "default" namespace by default

@afbjorklund
Copy link
Collaborator

I recall there was a lot of hassle with the CD-rom drivers with the initial arm64 initiative, but if it works OK in qemu2 too...

The minikube ISO has been updated since, so it is possible that the new OS works better (like with a new kernel or such?)

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2024
@afbjorklund
Copy link
Collaborator

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 7, 2024
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 18239) |
+-------------------+----------+---------------------+
| ⚠️  minikube start | 52.0s    | 72.7s ⚠️             |
| enable ingress    | 25.8s    | 25.8s               |
+-------------------+----------+---------------------+

Times for minikube start: 49.4s 52.2s 52.4s 51.3s 54.8s
Times for minikube (PR 18239) start: 164.3s 49.4s 50.4s 48.5s 51.2s

Times for minikube ingress: 26.5s 27.1s 26.7s 26.0s 22.6s
Times for minikube (PR 18239) ingress: 26.7s 26.6s 26.6s 25.6s 23.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18239) |
+----------------+----------+---------------------+
| minikube start | 24.3s    | 22.4s               |
| enable ingress | 20.4s    | 20.5s               |
+----------------+----------+---------------------+

Times for minikube ingress: 21.3s 19.8s 20.3s 20.3s 20.4s
Times for minikube (PR 18239) ingress: 21.3s 20.3s 20.3s 20.3s 20.3s

Times for minikube start: 26.1s 25.7s 22.1s 22.4s 25.3s
Times for minikube (PR 18239) start: 22.4s 22.3s 21.9s 22.7s 22.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 18239) |
+----------------+----------+---------------------+
| minikube start | 21.8s    | 21.9s               |
| enable ingress | 31.0s    | 34.1s               |
+----------------+----------+---------------------+

Times for minikube (PR 18239) start: 21.4s 21.9s 21.9s 23.5s 21.0s
Times for minikube start: 21.8s 21.3s 23.4s 21.0s 21.6s

Times for minikube ingress: 30.8s 30.8s 30.8s 31.8s 30.9s
Times for minikube (PR 18239) ingress: 31.3s 46.8s 30.8s 30.8s 30.8s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd_arm64 TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 2.91 (chart)
Docker_macOS TestMountStart/serial/VerifyMountPostStop (gopogh) 5.32 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 7.29 (chart)
Hyperkit_macOS TestDockerFlags (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/parallel/CpCmd (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/parallel/SSHCmd (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/CacheCmd/cache/verify_cache_inside_node (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/KubeContext (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/KubectlGetPods (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/LogsCmd (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/LogsFileCmd (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/MinikubeKubectlCmd (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/MinikubeKubectlCmdDirectly (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/SoftStart (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/serial/StartWithProxy (gopogh) 17.59 (chart)
Hyperkit_macOS TestImageBuild/serial/Setup (gopogh) 17.59 (chart)
Hyperkit_macOS TestJSONOutput/pause/Command (gopogh) 17.59 (chart)
Hyperkit_macOS TestJSONOutput/start/Command (gopogh) 17.59 (chart)
Hyperkit_macOS TestJSONOutput/start/parallel/DistinctCurrentSteps (gopogh) 17.59 (chart)
Hyperkit_macOS TestJSONOutput/start/parallel/IncreasingCurrentSteps (gopogh) 17.59 (chart)
Hyperkit_macOS TestJSONOutput/unpause/Command (gopogh) 17.59 (chart)
Hyperkit_macOS TestMultiNode/serial/MultiNodeLabels (gopogh) 17.59 (chart)
Hyperkit_macOS TestMultiNode/serial/ProfileList (gopogh) 17.59 (chart)
Hyperkit_macOS TestPause/serial/Start (gopogh) 17.59 (chart)
Hyperkit_macOS TestRunningBinaryUpgrade (gopogh) 17.59 (chart)
Hyperkit_macOS TestStoppedBinaryUpgrade/MinikubeLogs (gopogh) 17.59 (chart)
Hyperkit_macOS TestFunctional/parallel/UpdateContextCmd/no_changes (gopogh) 17.76 (chart)
Hyperkit_macOS TestFunctional/parallel/UpdateContextCmd/no_clusters (gopogh) 17.76 (chart)
Hyperkit_macOS TestMultiNode/serial/ValidateNameConflict (gopogh) 17.92 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@nirs
Copy link
Contributor Author

nirs commented Mar 7, 2024

Failing tests look similar to #18241 - change in libvirt domain xml cannot break other drivers.

@medyagh medyagh changed the title kvm2 driver: Use scsi cdrom for arm64 kvm2 driver: Use scsi cdrom for arm64 to fix Linux on Apple Silicone Mar 8, 2024
Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nirs thank you for this contribution, and happy to see that this will help users with linux on apple silicone.

I look forward to see more contributions from you

@medyagh medyagh merged commit 8000ca5 into kubernetes:master Mar 8, 2024
24 of 35 checks passed
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund, BlaineEXE, medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [afbjorklund,medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Minikube cluster using kvm2 driver is broken on linux/aarch64
7 participants