krunkit: Use unix socket for --restul-uri #20900
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Update to longterm kernel 6.6.92[1]
- aarch64: Enable Virtio GPU, needed for krunkit driver
Generated using by running:
make iso-menuconfig-aarch64
make linux-menuconfig-aarch64
make iso-menuconfig-x86_64
make linux-menuconfig-x86_64
This generated many changes in the configs, maybe they were updated
manually previously.
With this change we can boot krunkit with the built iso:
% minikube start -p krunkit --driver krunkit --container-runtime containerd --iso-url file://$PWD/minikube-arm64-vgpu.iso
😄 [krunkit] minikube v1.36.0 on Darwin 15.5 (arm64)
✨ Using the krunkit (experimental) driver based on user configuration
👍 Starting "krunkit" primary control-plane node in "krunkit" cluster
🔥 Creating krunkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
📦 Preparing Kubernetes v1.33.1 on containerd 1.7.23 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "krunkit" cluster and "default" namespace by default
And now we have accelerated gpu:
$ tree /dev/dri
/dev/dri
|-- by-path
| |-- platform-a007000.virtio_mmio-card -> ../card0
| `-- platform-a007000.virtio_mmio-render -> ../renderD128
|-- card0
`-- renderD128
For example usage of the accelerated GPU see:
https://github.com/medyagh/ai-playground-minikube/tree/main/macos
[1] https://www.kernel.org/
libkrun virtio-net driver enables TSO offloading and checksum offloading by default, so we must use vment-helper --enable-tso and --enable-checksum-offload with krunkit. These options do not work with vfkit.
krunkit is a tool to launch configurable virtual machines using the libkrun platform, optimized for GPU accelerated virtual machines and AI workloads on Apple silicon. It is mostly compatible with vfkit; the driver is a simplified copy of the vfkit driver. Unlike vfkit, krunkit is available only on Apple silicon. Changes compared to vfkit driver: - krunkit requires unix socket for netwroking, so we must use vment-helper. - krunkit can be controlled only via HTTP, not via unix socket. - krunkit does not support HardStop so we need to use SIGKILL. - krunkit does not support --kernel-cmdline - We must enable vmnet offloading, required for krunkit. - The code was simplified since vmnet-helper is always used - Code was cleaned up to use .ResolveStorePath() Limitations: - Only one machine can be created since we use the same port for krunkit --restful-uri. This should be fixed to use an unused port, or use a unix socket when unix socket is supported[1]. [1] containers/krunkit#47
Previously it was used only for vfkit, so we suggested to fallback to the `nat` network. This advice is not relevant to krunkit or to qemu (which can also use vmnet-helper). Change the error to recommend installing vment-helper. We need to think how we can recommend other networks for vfkit and qemu. Another solution is to create error for every driver+network combination but this seems hard to manage.
This is the same way that we test vfkit. This test is not running in the CI. Issues: - Need to install and configure vment-helper (requires root).
This removes the limit of using a single machine, removing the need to
manage tcp ports. We use:
--restful-uri unix://$MINIKUBE_HOKE/machines/NAME/krunkit.sock
The socket is created and deleted by krunkit.
NOTES:
- Depends on containers/krunkit#51.
To test this change build and install krunkit from this PR.
k8s-ci-robot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: nirs The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
|
Hi @nirs. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XXL
|
@nirs would you like ISO to be built for this PR ? (for you to test it ?) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This removes the limit of using a single machine, removing the need to
manage tcp ports. We use:
The socket is created and deleted by krunkit.
NOTES:
To test this change build and install krunkit from this PR.
Status:
Based on #20826 for testing.