MicroVM Support

MicroVM Support

scripts/microvm/run-microvm.sh boots an already-built OCI image layout directly under plain QEMU/KVM instead of inside a container - the same consumer role the Dockerfile plays for containers. Build the image first the normal way (cmd/oci-builder), then point this script at it.

Design invariant: the microVM always runs content from a genuine, verified OCI image. Every invocation independently re-verifies the layout it's handed with scripts/ci/verify-oci-layout.py - the same check CI itself trusts - before extracting a single byte of it into the initramfs. This isn't a convenience default; it cannot be skipped or disabled. An arbitrary hand-crafted directory, or a layout that's been tampered with, is refused outright, not booted.

Why a microVM instead of a container

A container shares the host kernel; a microVM boots its own. That difference is the entire reason this exists, for two distinct goals:

1. Maximum practical local isolation. Even a hardened container (--read-only, --cap-drop=ALL, no-new-privileges, seccomp) still shares the host's kernel attack surface - a kernel vulnerability can be a container escape. A microVM's guest kernel is a completely separate one; KVM's guest/host memory isolation is a much stronger boundary than Linux namespaces/cgroups. If "as secure as this project can make it, locally, with nothing exotic" is the goal, a microVM is the answer containers can't give you.
2. Running something that can't (or shouldn't) be containerized. A legacy binary that assumes it owns PID 1 in a very particular way, that expects a real (if minimal) kernel and not just namespaced views of the host one, or that you simply don't want sharing a kernel with anything else - the OCI layout is still the right packaging format (deterministic, minimal, non-root, no baked-in credentials), you just don't hand it to a container runtime. run-microvm.sh reads the image's actual declared architecture and entrypoint out of its own OCI config and boots that, so this works for any binary cmd/oci-builder accepts - static or, via -extra-file, dynamically linked - including ones this repository never built.

What you give up: boot time (seconds, not milliseconds), and this tooling deliberately does not implement cross-architecture emulation - see "Architecture dynamism" below.

What it deliberately doesn't use

• No Firecracker, no Cloud Hypervisor, no other VMM binary - just qemu-system-x86_64/qemu-system-aarch64, already-packaged on any major Linux distribution.
• No prebuilt kernel binary - scripts/microvm/build-kernel.sh downloads Linux kernel source from kernel.org (checksum-verified) and builds a minimal kernel image itself.
• No ext4 image, no loop device, no root-filesystem block device at all - the guest boots directly from a cpio initramfs built from the OCI layer's own extracted content.
• No TAP device, no bridge, no root/CAP_NET_ADMIN, no DHCP client - networking is QEMU's unprivileged user-mode ("SLIRP") backend with a static guest IP set by the Linux kernel's own built-in ip= autoconfiguration.
• No new Go module dependency - go.mod is unchanged; cmd/microvm-init is stdlib-only.

Architecture dynamism

This works on both amd64 (x86_64) and arm64 (aarch64) hosts. KVM only accelerates a guest matching the host architecture - there is no software-emulated (TCG) fallback here by design, so an amd64 image can't boot on an arm64 host or vice versa. run-microvm.sh checks the image's declared architecture against the host and fails clearly, rather than silently falling back to slow emulation that would undercut the whole point of using KVM.

	amd64 host	arm64 host
QEMU binary	qemu-system-x86_64	qemu-system-aarch64
Machine type	microvm,pcie=on,acpi=on	virt
Console UART	ttyS0 (8250)	ttyAMA0 (PL011)
PCI enumeration	ACPI-described	device-tree-described (QEMU auto-generates the DT)

Both use the same virtio-net-pci device model over that machine's PCI topology - the only real divergence is the console driver and how PCI is described to the guest. See scripts/microvm/lib-arch.sh (the shared uname -m → arch/QEMU-binary mapping every script sources) and scripts/microvm/kernel-common.config / scripts/microvm/kernel-<arch>.config (the shared vs. arch-specific kernel config fragments).

Installing KVM

Run scripts/microvm/check-kvm.sh first - a non-destructive, no-sudo check of exactly which of the steps below you still need. It works on both architectures.

1. Confirm virtualization support. grep -Em1 'vmx|svm' /proc/cpuinfo (amd64 only - arm64 has no single universal cpuinfo flag for this; check-kvm.sh falls back to checking /dev/kvm directly there). If nothing prints on amd64, enable Intel VT-x/AMD-V in firmware, or request a nested-virtualization-capable instance type on a cloud/virtual host.
2. Install KVM and QEMU.
   • Debian/Ubuntu (amd64): sudo apt-get install -y qemu-system-x86 cpio build-essential flex bison bc libssl-dev libelf-dev
   • Debian/Ubuntu (arm64): sudo apt-get install -y qemu-system-arm cpio build-essential flex bison bc libssl-dev libelf-dev
   • Fedora/RHEL/CentOS Stream: sudo dnf install -y qemu-kvm cpio gcc make flex bison bc openssl-devel elfutils-libelf-devel
3. Grant your user /dev/kvm access. sudo usermod -aG kvm "$(whoami)", then log out/in (or newgrp kvm).
4. Verify. scripts/microvm/check-kvm.sh - every line should print OK. If /dev/kvm still isn't accessible, confirm the module is loaded (lsmod | grep kvm; sudo modprobe kvm_intel/kvm_amd on amd64, sudo modprobe kvm on arm64) and that you started a new login session after the group change.

Running it

# 1. Build a layout the normal way.
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -o /tmp/service ./cmd/example-service
go run ./cmd/oci-builder -binary /tmp/service -output ./oci-image -arch amd64

# 2. Boot it.
scripts/microvm/run-microvm.sh ./oci-image        # port 8080 by default
scripts/microvm/run-microvm.sh ./oci-image 9090   # or a custom port

run-microvm.sh:

1. Independently re-verifies OCI_IMAGE_DIR with scripts/ci/verify-oci-layout.py - the same from-scratch, dependency-free check CI runs on every layout it trusts (digests, descriptor structure, and, critically before anything is extracted, that the layer's tar contains no unsafe paths, symlinks, or hardlinks). This is not optional and cannot be skipped or overridden by an environment variable: the microVM must always boot content that genuinely came from a cmd/oci-builder layout, never an arbitrary hand-crafted directory. A directory that isn't a real, structurally valid layout is refused here, before anything else runs.
2. Reads the (now-verified) image's architecture and Entrypoint straight from its OCI config and refuses to continue if the architecture doesn't match the host.
3. Builds cmd/microvm-init for that architecture.
4. Extracts the image's own (verified) layer, adds cmd/microvm-init as /sbin/init, packs it all into a cpio initramfs.
5. Builds (or reuses a cache at .cache/microvm/<arch>/kernel) the from-source kernel.
6. Boots under KVM, waits for the entrypoint to start listening on the requested port (a protocol-agnostic TCP check, not an HTTP assumption - this has to work for arbitrary "legacy app" binaries), then streams the guest's console live.
7. Stays running in the foreground - Ctrl-C stops and cleans up. On a boot failure, prints the full console log.

Environment overrides (see run-microvm.sh with no arguments for the full, current list): MICROVM_MEMORY (default 128M), MICROVM_SMP (default 1), MICROVM_KERNEL (point at a prebuilt kernel instead of building one), MICROVM_QEMU_SANDBOX (see "Security hardening" below), MICROVM_SMOKE_TEST_PATH (curl one path once and exit instead of staying up - what CI uses for a boot smoke test), MICROVM_SHUTDOWN_TEST_PATH (POST one path once, then wait for the guest to power itself off and verify QEMU exits cleanly - a graceful-shutdown test, not a boot smoke test; see "Scope, provenance, and evidence" below), and MICROVM_BOOT_MANIFEST (write a combined kernel+init+layer digest manifest to this path).

The kernel build is the slow step (minutes, not seconds, even minimal). Delete .cache/microvm/<arch>/kernel, or set FORCE_REBUILD=1, to force a rebuild after changing scripts/microvm/kernel-common.config or scripts/microvm/kernel-<arch>.config.

How boot works, precisely

sequenceDiagram
    participant You
    participant Script as run-microvm.sh
    participant Verify as verify-oci-layout.py
    participant QEMU
    participant Kernel
    participant Init as cmd/microvm-init (PID 1)
    participant App as your entrypoint

    You->>Script: ./oci-image [port]
    Script->>Verify: is this a real layout?
    Verify-->>Script: OCI_VALIDATION_OK (or refuse)
    Script->>Script: build init + assemble initramfs
    Script->>QEMU: boot (-kernel, -initrd, -append)
    QEMU->>Kernel: start
    Kernel->>Init: rdinit=/sbin/init
    Init->>App: exec (argv after --)
    App-->>You: listening on the forwarded port
    You->>App: curl / smoke test
    App-->>Init: exits
    Init->>Kernel: syscall.Reboot(POWER_OFF)

Loading

1. cmd/microvm-init sits at /sbin/init in the initramfs; the kernel boots with rdinit=/sbin/init - straight from the initramfs, no root filesystem switch/pivot at all.
2. The image's declared entrypoint is appended to the kernel command line after a literal --, which the kernel forwards verbatim as /sbin/init's argv (a standard, documented Linux kernel convention: any command-line words the kernel itself doesn't recognize become init's arguments). microvm-init, as PID 1, execs that path, forwards SIGTERM/SIGINT to it, waits for it to exit, then calls syscall.Reboot(LINUX_REBOOT_CMD_POWER_OFF) to cleanly stop the VM. It does not reap unrelated reparented orphans - the image runs exactly one process, so general-purpose init duties (the kind a real init(1) or tini handles) are intentionally out of scope. See Scripts Reference for how this is tested without ever invoking the real, irreversible poweroff syscall in a test.
3. Networking is one virtio-net-pci device on QEMU's SLIRP backend, with a static address (10.0.2.15, SLIRP's well-known default guest address) set via the kernel's own ip= boot parameter - there's no DHCP client in the image, so this has to be static.

Security hardening

Beyond KVM's own guest/host isolation, run-microvm.sh locks the host QEMU process itself down:

• -no-user-config - ignore any host QEMU config files.
• -monitor none - no QEMU control-plane socket/console at all; this is a fully automated, non-interactive boot with nothing to attach to.
• -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny - QEMU's own seccomp self-sandboxing, restricting which syscalls the QEMU host process can make. This is defense in depth against a QEMU VMM-escape bug specifically - something KVM's memory isolation alone doesn't cover, since it protects the guest from a compromised QEMU, not the host from one.

-sandbox on requires QEMU built with seccomp support (true of the standard Debian/Ubuntu/Fedora packages). If your build lacks it, run-microvm.sh fails fast with a clear error from QEMU itself rather than silently degrading - disable it with MICROVM_QEMU_SANDBOX= scripts/microvm/run-microvm.sh ... if needed.

The guest's writable state is entirely memory-backed initramfs, which vanishes the moment the VM powers off - there is no persistent disk to leave anything behind on.

Scope, provenance, and evidence

Interpreting macOS/HVF evidence

ci-microvm.yml always exercises the native Darwin implementation, but a green hosted macOS job has two possible meanings. When the runner exposes HVF, the signed TestRunLinuxWithRealHVF and TestDarwinVMMWithRealHVF tests provide real hardware boot and lifecycle evidence. When Virtualization.framework reports that virtualization is unavailable on the nested runner, CI excludes exactly those two hardware tests and still requires the complete Darwin contract suite to pass. That second outcome is continuous contract evidence, not a hardware boot claim.

For an actual HVF proof, run scripts/microvm/test-hvf-local.sh on Apple Silicon without SECURE_OCI_ALLOW_UNAVAILABLE_HVF=1. The local script is fail-closed, uses the local ARM64 kernel/initramfs cache, entitlement-signs the test binary and succeeds only if both hardware tests pass. The hosted job sets the fallback flag solely because nested virtualization availability is outside the repository's control.

The kernel and initramfs are deliberately not part of the signed OCI artifact - only the already-verified OCI layout is a distributable, signed, content-addressed artifact (see GHCR/Cosign/Kubernetes demonstration). The kernel is built from a version-pinned, checksum-verified upstream source on demand, and the initramfs is assembled fresh at every boot from that layout's own layer plus a freshly built cmd/microvm-init; neither is ever written to persistent storage. This keeps the trust root at the signed OCI image and treats the kernel/initramfs as reproducible, disposable build output rather than a second thing to ship and sign - see Threat Model and Residual Risks for the full reasoning (T15, T18, T20).

ci-microvm.yml still produces real evidence about that disposable kernel, on every push and pull request:

• Provenance. build-kernel.sh writes kernel.provenance.json next to the kernel image: version, upstream source URL, source tarball checksum, the built kernel image's own SHA-256, and the resolved .config's SHA-256. It lands in the same cache directory as the kernel, so a CI cache hit carries it forward without rebuilding.
• SBOM. build-kernel.sh also writes kernel.sbom.cdx.json, a real CycloneDX 1.5 document with one operating-system component for the guest kernel (name, version, SHA-256 hash, source distribution URL, resolved-config and source-archive checksums as CycloneDX properties). Its serialNumber is a deterministic UUIDv5 derived from the source URL and kernel hash, not a random one or a wall-clock timestamp - either would otherwise make two builds of the identical kernel produce a different SBOM, which would undercut the reproducibility this whole pipeline cares about. This is the "guest OS" half of an SBOM; the application half is the existing BuildKit SBOM on the OCI image itself (see GHCR/Cosign/Kubernetes demonstration).
• Hardening scan, and real hardening from it. kernel-hardening-checker (formerly kconfig-hardened-check) runs against the resolved .config and is uploaded as both a human-readable and a JSON report. This is informational, not a hard gate: it's a general-purpose Linux hardening baseline, and this kernel is deliberately minimal and single-purpose, so a real fraction of its checks don't apply. Its output has already driven real changes in kernel-common.config though: CONFIG_DEVMEM, CONFIG_DEVPORT, CONFIG_KEXEC(_FILE), CONFIG_HIBERNATION, CONFIG_CRASH_DUMP, CONFIG_FB, CONFIG_LEGACY_PTYS/_TIOCSTI, CONFIG_BPF_SYSCALL, CONFIG_MAGIC_SYSRQ, and CONFIG_DEBUG_FS are disabled, and CONFIG_SYN_COOKIES is enabled - a verified 99-to-81 FAIL reduction with no side effects (every disabled option was confirmed to already be a plain =y bool, not a tristate =m, before touching it). One change was tried and reverted: CONFIG_MODULES=n looked like an obvious win (no modprobe or /lib/modules exist in this image), but on this general-purpose defconfig it silently promotes hundreds of previously-=m drivers (netfilter conntrack, IPv6, cpufreq governors, ...) straight to =y - permanently compiled in and active - since Kconfig can no longer leave them as loadable modules. That's a net increase in attack surface, not a reduction, so CONFIG_MODULES is deliberately left at the defconfig default; see the comment above it in kernel-common.config for the full reasoning. CONFIG_VT, CONFIG_IO_URING, and CONFIG_KALLSYMS were also tried and found to be silently re-enabled by Kconfig regardless (pulled back in by the large DRM/perf/ftrace surface this defconfig already turns on) - removing them for real would mean trimming that surface directly, out of scope here.
• Combined boot-bundle digest. Every run-microvm.sh invocation - not just CI's - computes and logs a manifest tying together the image's architecture and entrypoint, the OCI layer digest, the cmd/microvm-init binary's SHA-256, the kernel image's SHA-256, the kernel provenance record above, and a single combined_digest SHA-256 over all of those - one digest that covers everything actually booted for a given run, even though the kernel and initramfs themselves aren't part of the OCI manifest. Set MICROVM_BOOT_MANIFEST=path to also persist it outside the run's temporary directory (what CI does).
• Initramfs reproducibility. scripts/microvm/assemble-initramfs.sh (extracted from run-microvm.sh so CI can call it twice) is byte-for-byte reproducible given the same layer and init binary: it uses tar --delay-directory-restore on extraction (without it, a directory that later receives a freshly-added file - like /sbin - keeps the wall-clock extraction time instead of the layer's own fixed timestamps) and cpio --reproducible (zeroes the device/inode fields cpio's newc format otherwise bakes in from the extracting filesystem) over a fixed, sorted entry order. ci-microvm.yml assembles it twice independently and diffs the bytes on every run.
• Kernel image reproducibility: a known, unresolved gap. Unlike the initramfs above, the kernel image itself is not currently proven byte-reproducible, and this has been verified rather than assumed: two independent builds from identical inputs (source, config, toolchain) produced different bytes, isolated to a small (~20-byte) region. KBUILD_BUILD_TIMESTAMP/_USER/_HOST and SOURCE_DATE_EPOCH are set in build-kernel.sh (standard practice, and correct regardless), and a .note.gnu.build-id was suspected and directly tested as the cause - disproven: removing it (--build-id=none) produced byte-for-byte identical output to not removing it, because arch/arm64/boot/Image is produced via objcopy -O binary, which only extracts loadable (PT_LOAD) segments, and ELF notes live outside any of those. The real remaining source hasn't been root-caused. Kernel provenance is still attested (source/config/image checksums, signed), so a specific build's identity is fully verifiable either way - what isn't yet true is that rebuilding reproduces the same bytes, the way the OCI layer and the initramfs both do.
• Signed evidence bundle. On push to main only (never on pull_request, even from a fork that still gets a real OIDC token - the same reasoning ci-release.yml applies to signing the OCI image itself), a separate sign-kernel-evidence job merges the provenance record, the CycloneDX SBOM, the hardening report, and a boot-bundle manifest into one file and signs it with keyless Cosign, verifying the result with the same --certificate-identity-regexp/--certificate-oidc-issuer pattern used for the image.
• Graceful shutdown, tested end to end. Beyond the unit tests covering cmd/microvm-init's signal-forwarding logic in isolation (see Scripts Reference), ci-microvm.yml boots the image a second time with MICROVM_SHUTDOWN_TEST_PATH=/debug/exit (a POST-only debug endpoint on cmd/example-service that responds then exits(0) on its own). run-microvm.sh then waits for QEMU to exit on its own rather than being killed, checks its exit status is 0, and greps the console log for cmd/microvm-init's own action=poweroff line - proving the full real path (app exits → init observes the exit → init calls syscall.Reboot(POWER_OFF) → QEMU shuts down cleanly), not just the unit-tested pieces of it.

Two further scope limits are intentional, not oversights, beyond what's already covered above and in "What it deliberately doesn't use": there is no volume or persistent-storage mechanism for the microVM guest at all (see "Security hardening" above - its entire writable state is the memory-backed initramfs), and the boot path targets exactly one VMM (QEMU/KVM) with no support for Firecracker, Cloud Hypervisor, or any other microVM launcher or disk-image format. The OCI layout it consumes is the portable, launcher-agnostic artifact (see Dockerfile Consumer); the microVM boot script itself is not.

Troubleshooting

See Troubleshooting for the consolidated list.