iso: Update to longterm kernel to 6.6.95 #20995

nirs · 2025-06-29T18:40:55Z

Update linux to longterm kernel 6.6.95. This provides VirtIO GPU support required for kerunkit driver to enable AI workload on macOS.

The update was generated using iso-menuconfig and linux-menuconfig, and there are no manual changes in the configuration files. Future updates should be easy if we continue to use kconfig.

buildroot supports also longterm kernel 6.12.35 but we know that we have issues with newer kernels and Rosetta in macOS 15[1]. We know that kernel 6.13+ is broken and kernel 6.8 is ok. Since Rosetta is the most important feature for me and time is limited, I'm going with kernel 6.6 for now.

Status

✅ Building iso locally
arm64:
- ✅ vfkit - - tested with ramen rook testing environment: https://github.com/RamenDR/ramen/blob/main/test/envs/rook.yaml
- ✅ qemu - starting cluster with socket_vmnet network
- ✅ krunkit - starting cluster using Add krunkit driver supporting GPU acceleration on macOS #20826
x86_64:
- ✅ kvm2 - tested with ramen regional dr testing environment: https://github.com/RamenDR/ramen/blob/main/test/envs/regional-dr.yaml

[1] lima-vm/lima#3592

k8s-ci-robot · 2025-06-29T18:41:05Z

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nirs · 2025-06-29T19:56:29Z

ok-to-build-iso

nirs · 2025-06-29T20:02:24Z

Testing iso

arm64

vfkit

% minikube start --driver vfkit --network vmnet-shared --container-runtime containerd --iso-url file://$PWD/minikube-arm64-6.6-vgpu.iso
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the vfkit driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

% minikube ssh -- uname -a
Linux minikube 6.6.95 #2 SMP PREEMPT Sun Jun 29 17:39:58 UTC 2025 aarch64 GNU/Linux

qemu

% minikube start --driver qemu2 --container-runtime containerd --iso-url file://$PWD/minikube-arm64-6.6-vgpu.iso
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating qemu2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

% out/minikube ssh -- uname -a
Linux minikube 6.6.95 #2 SMP PREEMPT Sun Jun 29 17:39:58 UTC 2025 aarch64 GNU/Linux

krunkit

% out/minikube start --driver krunkit --container-runtime containerd --iso-url file://$PWD/../minikube-arm64-6.6-vgpu.iso
😄  minikube v1.36.0 on Darwin 15.5 (arm64)
✨  Using the krunkit (experimental) driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating krunkit VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

% out/minikube ssh -- uname -a
Linux minikube 6.6.95 #2 SMP PREEMPT Sun Jun 29 17:39:58 UTC 2025 aarch64 GNU/Linux

% out/minikube ssh -- tree /dev/dri
/dev/dri
|-- by-path
|   |-- platform-a007000.virtio_mmio-card -> ../card0
|   `-- platform-a007000.virtio_mmio-render -> ../renderD128
|-- card0
`-- renderD128

X86_64

kvm2

$ minikube start --driver kvm2 --container-runtime containerd --iso-url file:///$PWD/minikube-amd64-6.6-1.iso
😄  minikube v1.36.0 on Fedora 42 (kvm/amd64)
✨  Using the kvm2 driver based on user configuration
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating kvm2 VM (CPUs=2, Memory=6144MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.33.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube ssh -- uname -a
Linux minikube 6.6.95 #1 SMP PREEMPT_DYNAMIC Sun Jun 29 15:29:19 UTC 2025 x86_64 GNU/Linux

minikube-bot · 2025-06-30T02:51:19Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

ComradeProgrammer · 2025-06-30T12:25:54Z

/ok-to-test

medyagh · 2025-06-30T17:08:10Z

/retest-this-please

nirs · 2025-07-02T08:40:04Z

ok-to-build-iso

nirs · 2025-07-02T09:00:55Z

@medyagh not sure what is the issue with the Jenkins Cross Build.

Here is a successful build from my fork:
https://github.com/nirs/minikube/actions/runs/16020440408/job/45195869925

I also run locally make cross without any issues, using go 1.24.4:

% make cross
GOOS="linux" GOARCH="amd64"  \
        go build -tags "" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.36.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.36.0-1751315722-20991 -X k8s.io/minikube/pkg/version.gitCommitID="8fba56237f4b719356286263873b210ed2245bb6" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-linux-amd64 k8s.io/minikube/cmd/minikube
GOOS="darwin" GOARCH="amd64"  \
        go build -tags "" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.36.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.36.0-1751315722-20991 -X k8s.io/minikube/pkg/version.gitCommitID="8fba56237f4b719356286263873b210ed2245bb6" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-darwin-amd64 k8s.io/minikube/cmd/minikube
GOOS="windows" GOARCH="amd64"  \
        go build -tags "" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.36.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.36.0-1751315722-20991 -X k8s.io/minikube/pkg/version.gitCommitID="8fba56237f4b719356286263873b210ed2245bb6" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-windows-amd64 k8s.io/minikube/cmd/minikube
cp out/minikube-windows-amd64 out/minikube-windows-amd64.exe

nirs · 2025-07-02T09:06:14Z

@medyagh ok, it seems that the hyperkit build is failing:

GOOS=darwin CGO_ENABLED=1 go build \
	-ldflags="-X k8s.io/minikube/pkg/drivers/hyperkit.version=v1.36.0 -X k8s.io/minikube/pkg/drivers/hyperkit.gitCommitID="c7ff6c486f85ff5c4dd71556f80d344fa4e33ccb""   \
	-o out/docker-machine-driver-hyperkit k8s.io/minikube/cmd/drivers/hyperkit
+ failed=2

Maybe some issue with the xcgo image?

nirs · 2025-07-02T09:28:46Z

Added #21012 for dropping hyperkit.

minikube-bot · 2025-07-02T11:51:35Z

Hi @nirs, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

nirs · 2025-07-02T12:23:38Z

ok-to-test

ComradeProgrammer · 2025-07-02T12:31:20Z

/ok-to-test

medyagh · 2025-07-02T17:30:15Z

hm... the jenkins build error
https://storage.googleapis.com/minikube-builds/logs/20995/build.txt

thats werid that xcgo build thing suddenly decided to fail, I wonder what changed
how serious is the xcgo issue, is that salvagable ?

(feel free to Comment the hyperkit in your PR and we can decide later) we can focus on kernel6 for now)

medyagh · 2025-07-02T17:30:32Z

/retest-this-please

medyagh · 2025-07-03T20:42:46Z

/retest-this-please

nirs · 2025-07-05T13:13:01Z

@medyagh @ComradeProgrammer tests seems stuck for few days. Maybe we need a /retest-this-please or do some cleanup on Jenkins?

Generated by running `make iso-menuconfig-x86_64` and updating kernel version to longterm kernel 6.6.95 and kernel headers to 6.6.x, and then running `make linux-menuconfig-x86_64` to update the linux config. Additinally update hyperv-daemons package to use kernel 6.x.

Generated by running `make iso-menuconfig-aarch64` and updating kernel version to longterm kernel 6.6.95 and kernel headers to 6.6.x, and then running `make linux-menuconfig-aarch64` to update the linux config.

The krunkit driver exposes the host GPU via VirtIO GPU, enabling AI workloads in the guest.

nirs · 2025-07-05T17:58:39Z

I rebase on master to consume #21027

medyagh · 2025-07-07T20:07:36Z

/retest-this-please

medyagh · 2025-07-08T17:55:19Z

/retest-this-please

minikube-pr-bot · 2025-07-08T18:43:02Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20995) |
+----------------+----------+---------------------+
| minikube start | 49.8s    | 52.3s               |
| enable ingress | 15.6s    | 14.9s               |
+----------------+----------+---------------------+

Times for minikube start: 49.8s 49.6s 49.2s 51.2s 49.2s
Times for minikube (PR 20995) start: 51.9s 52.4s 50.2s 52.4s 54.4s

Times for minikube (PR 20995) ingress: 15.0s 15.0s 15.2s 14.5s 15.0s
Times for minikube ingress: 16.0s 16.0s 15.0s 15.5s 15.5s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20995) |
+----------------+----------+---------------------+
| minikube start | 23.8s    | 25.1s               |
| enable ingress | 13.3s    | 13.2s               |
+----------------+----------+---------------------+

Times for minikube start: 24.9s 23.1s 25.4s 22.7s 22.9s
Times for minikube (PR 20995) start: 24.9s 24.0s 26.1s 24.3s 26.1s

Times for minikube ingress: 12.3s 13.3s 13.8s 13.3s 13.8s
Times for minikube (PR 20995) ingress: 12.8s 13.3s 13.9s 12.8s 13.3s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 20995) |
+----------------+----------+---------------------+
| minikube start | 22.9s    | 23.6s               |
| enable ingress | 26.9s    | 29.3s               |
+----------------+----------+---------------------+

Times for minikube start: 22.2s 22.0s 23.8s 21.8s 24.7s
Times for minikube (PR 20995) start: 22.7s 22.7s 23.2s 27.3s 22.3s

Times for minikube ingress: 23.8s 39.8s 24.3s 23.8s 22.8s
Times for minikube (PR 20995) ingress: 22.9s 22.8s 38.8s 22.8s 39.3s

medyagh · 2025-07-08T20:06:49Z

/lgtm

k8s-ci-robot · 2025-07-08T20:06:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, nirs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 29, 2025

k8s-ci-robot requested review from ComradeProgrammer and medyagh June 29, 2025 18:41

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 29, 2025

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 29, 2025

This was referenced Jun 29, 2025

WIP: Update kernel to 6.6 #20923

Closed

Add krunkit driver supporting GPU acceleration on macOS #20826

Open

nirs mentioned this pull request Jun 29, 2025

ISO: update linux kernel version to 6.12.19 #20978

Closed

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 30, 2025

This comment has been minimized.

Sign in to view

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 2, 2025

nirs force-pushed the iso-linux-6.6 branch from 939bf76 to 8fba562 Compare July 2, 2025 08:04

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 2, 2025

nirs marked this pull request as ready for review July 2, 2025 08:05

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 2, 2025

k8s-ci-robot requested a review from prezha July 2, 2025 08:05

This comment has been minimized.

Sign in to view

nirs and others added 4 commits July 5, 2025 20:57

iso: Update kernel to 6.6.95 for aarch64

9420cd6

Generated by running `make iso-menuconfig-aarch64` and updating kernel version to longterm kernel 6.6.95 and kernel headers to 6.6.x, and then running `make linux-menuconfig-aarch64` to update the linux config.

iso: Enable VirtIO GPU for krunkit driver

f94e996

The krunkit driver exposes the host GPU via VirtIO GPU, enabling AI workloads in the guest.

Updating ISO to v1.36.0-1751445739-20995

d65387f

nirs force-pushed the iso-linux-6.6 branch from 28b3856 to d65387f Compare July 5, 2025 17:57

This comment has been minimized.

Sign in to view

rata mentioned this pull request Jul 7, 2025

ISO upgrades to support user namespaces #18488

Closed

This comment has been minimized.

Sign in to view

medyagh merged commit 4471135 into kubernetes:master Jul 8, 2025
24 of 34 checks passed

k8s-ci-robot assigned medyagh Jul 8, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 8, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 8, 2025

nirs deleted the iso-linux-6.6 branch July 8, 2025 20:09

iso: Update to longterm kernel to 6.6.95 #20995

iso: Update to longterm kernel to 6.6.95 #20995

Conversation

nirs commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

Uh oh!

k8s-ci-robot commented Jun 29, 2025

Uh oh!

nirs commented Jun 29, 2025

Uh oh!

nirs commented Jun 29, 2025

Testing iso

arm64

vfkit

qemu

krunkit

X86_64

kvm2

Uh oh!

minikube-bot commented Jun 30, 2025

Uh oh!

ComradeProgrammer commented Jun 30, 2025

Uh oh!

medyagh commented Jun 30, 2025

Uh oh!

This comment has been minimized.

nirs commented Jul 2, 2025

Uh oh!

nirs commented Jul 2, 2025

Uh oh!

nirs commented Jul 2, 2025

Uh oh!

nirs commented Jul 2, 2025

Uh oh!

minikube-bot commented Jul 2, 2025

Uh oh!

nirs commented Jul 2, 2025

Uh oh!

ComradeProgrammer commented Jul 2, 2025

Uh oh!

medyagh commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

medyagh commented Jul 2, 2025

Uh oh!

medyagh commented Jul 3, 2025

Uh oh!

This comment has been minimized.

nirs commented Jul 5, 2025

Uh oh!

This comment has been minimized.

nirs commented Jul 5, 2025

Uh oh!

This comment has been minimized.

medyagh commented Jul 7, 2025

Uh oh!

This comment has been minimized.

medyagh commented Jul 8, 2025

Uh oh!

minikube-pr-bot commented Jul 8, 2025

Uh oh!

medyagh commented Jul 8, 2025

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 8, 2025

Uh oh!

Uh oh!

nirs commented Jun 29, 2025 •

edited

Loading

medyagh commented Jul 2, 2025 •

edited

Loading