feat(gateway): OpenShell gateway microVM with libkrun#100
Draft
Conversation
|
All contributors have signed the DCO ✍️ ✅ |
drew
changed the title
drew
force-pushed
the
openclaw-vm/dn
branch
2 times, most recently
from
58d7858 to
6e6759e
Compare
Add a new navigator-vm library crate that boots k3s inside a libkrun microVM, accessible from the host via gvproxy port forwarding. Key components: - FFI bindings to libkrun C API (krun_create_ctx, krun_add_net_unixgram, etc.) - VmConfig with gateway() preset for k3s and custom exec mode - gvproxy integration: virtio-net via unixgram, DHCP, native HTTP port forwarding - gateway-init.sh: PID 1 init script with DHCP via udhcpc, mounts, k3s exec - build-rootfs.sh: builds Ubuntu 22.04 arm64 rootfs with k3s + busybox-static - Kubeconfig auto-extraction to ~/.kube/gateway.yaml - CLI integration as 'ncl gateway' with --exec, --port, --net flags - macOS codesigning and DYLD_FALLBACK_LIBRARY_PATH in ncl wrapper
Enable full NemoClaw control plane deployment inside the libkrun microVM so e2e tests can run against the VM instead of Docker. Build-time (build-rootfs.sh): - Package helm chart and inject into k3s static charts directory - Copy HelmChart CR and agent-sandbox manifests into rootfs - Pull and save arm64 container images as tarballs for airgap boot Boot-time (gateway-init.sh): - Enable flannel CNI (remove --flannel-backend=none and related flags) - Deploy bundled manifests to k3s auto-deploy directory - Patch HelmChart CR for VM context (pullPolicy, SSH placeholders) - Ensure DNS fallback when DHCP doesn't configure resolv.conf Post-boot (lib.rs): - Wait for navigator namespace created by Helm controller - Generate PKI and apply TLS secrets via host kubectl - Store cluster metadata and mTLS creds for CLI/SDK access - Set 'gateway' as active cluster for e2e test discovery Also bump VM to 8GB RAM / 4 vCPUs, add port 30051 forwarding, fix nemoclaw wrapper fingerprint to include navigator-vm crate, and add test:e2e:vm mise task.
Stop deleting meta.db in gateway-init.sh and include the native snapshotter, content store, and metadata DB in the rootfs built by build-rootfs.sh. Without meta.db, containerd re-extracts all image layers on every boot (~2 min for navigator/server on virtio-fs), causing kubelet CreateContainer timeouts. Also replace the etcd-snapshot approach with direct SQLite cleanup of the kine DB to remove stale pod/event/lease records.
Move the gateway VM launching out of `nemoclaw gateway` into its own `gateway` binary built from the navigator-vm crate. The nemoclaw CLI no longer links against libkrun or requires macOS hypervisor codesigning. Add scripts/bin/gateway wrapper (build + codesign + exec) and clean up scripts/bin/nemoclaw to remove navigator-vm artifacts.
Two #[ignore] tests that require libkrun + pre-built rootfs: - gateway_boots_and_service_becomes_reachable: starts the full gateway and verifies the gRPC service on port 30051 - gateway_exec_runs_guest_command: runs /bin/true inside the VM via --exec and checks the exit code
drew
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
openshell-vmlibrary crate that boots k3s inside a libkrun microVM on macOS ARM64gatewaybinary with gvproxy networking, DHCP, and port forwarding~/.kube/gateway.yamlfor immediatekubectlaccessDetails
Boots a full k3s Kubernetes cluster inside an Apple Hypervisor.framework microVM via libkrun. Uses gvproxy for user-mode networking (virtio-net) with DHCP, bypassing TSI which is incompatible with k3s loopback connections. Preserves containerd metadata across boots for fast startup.
New files
crates/openshell-vm/— library crate with libkrun FFI bindings,VmConfig,launch()crates/openshell-vm/src/main.rs— standalonegatewaybinarycrates/openshell-vm/scripts/— rootfs build script, init script, debug helpersscripts/bin/openshell— updated with codesigning andDYLD_FALLBACK_LIBRARY_PATHUsage
Prerequisites
brew tap slp/krun && brew install libkrun/opt/podman/bin/gvproxy)