mini-firecracker

A toy Firecracker written from scratch in Go. Boots a real Linux kernel via PVH, runs an interactive shell over virtio-blk, talks to the host over virtio-net, and snapshots/restores itself in under 300 ms.

Linux + KVM + x86_64 only. About 5,300 lines of Go, one external dependency (golang.org/x/sys).

Built as a learning artifact: every package is one concern, every phase a runnable milestone. The point is to make the KVM ioctl surface and Firecracker's architecture click from the inside, the same way mini-sentry does for gVisor.

---

Demo

Snapshot a guest mid-boot, resume it in another process, and poke the shell + network:

# Take a snapshot 800 ms into the boot.
$ ./mini-fc run \
      --kernel testdata/vmlinux-5.10.245 --mem 512 \
      --drive path=ubuntu-24.04.squashfs,ro \
      --net   tap=minifc0 \
      --cmdline "console=ttyS0 reboot=k panic=1 root=/dev/vda rootfstype=squashfs ro init=/bin/sh" \
      --snapshot-after 800ms --snapshot-to /tmp/snap
[mini-fc] snapshot written to /tmp/snap

# Restore + interact.
$ (sleep 1.5; printf '
    ip link set eth0 up
    ip addr add 172.16.0.2/30 dev eth0
    ping -c 3 172.16.0.1
    exit
  ') | ./mini-fc restore --from /tmp/snap
[mini-fc] restored from /tmp/snap in 262 ms
# ping -c 3 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=0.264 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.148 ms
3 packets transmitted, 3 received, 0% packet loss

That's a complete VM lifecycle — kernel boot, virtio-blk root mount, userspace shell, virtio-net up, snapshot, restore, continue — running through ~5 KB of state.json and a 512 MiB memory.bin.

Phases

Each phase is a runnable milestone. All five are landed.

#	Goal
0	host capability probe (mini-fc check)
1	PVH boot to kernel banner + panic
2	virtio-MMIO + virtio-blk + interactive shell
3	virtio-net via tap, host↔guest ping
4	snapshot / restore (vCPU + VM + virtio + RAM)

Restore times on AMD Ryzen + Debian 6.12.74:

• 128 MiB Phase-1 boot: 65–77 ms (under the sub-100 ms target)
• 512 MiB Phase-2/3 with virtio devices: 260–290 ms (dominated by memory.bin I/O — mmap(MAP_PRIVATE, fd) would drop this to single-digit ms)

Build

The Makefile cross-compiles from macOS and deploys via ssh; on Linux it builds natively.

make build           # CGO_ENABLED=0 go build -o mini-fc ./cmd/mini-fc
make check-host      # build + run `mini-fc check` locally
make deploy          # cross-compile, scp to $REMOTE, run check there

Pre-flight on the Linux host:

• /dev/kvm exists and your user is in the kvm group
• Go 1.22+ if building on the host
• For Phase 3 / 4 networking, one-time tap setup (needs sudo):

  sudo ip tuntap add dev minifc0 mode tap user $USER
  sudo ip addr add 172.16.0.1/30 dev minifc0
  sudo ip link set minifc0 up

mini-fc check confirms /dev/kvm accessibility and the KVM capabilities the VMM relies on:

$ ./mini-fc check
mini-fc host check — maxbox (linux/amd64)
  [ok]  /dev/kvm                accessible (fd=3)
  [ok]  KVM_GET_API_VERSION     12
  [ok]  KVM_CAP_USER_MEMORY     yes
  [ok]  KVM_CAP_IRQCHIP         yes
  [ok]  KVM_CAP_HLT             yes
  [ok]  KVM_CAP_IMMEDIATE_EXIT  yes
  [ok]  KVM_CAP_XSAVE2          yes
  [ok]  KVM_CAP_MAX_VCPUS       4096
  ...
Host OK for mini-firecracker.

Subcommands

mini-fc check                probe /dev/kvm + required KVM capabilities
mini-fc run     [flags]      boot a PVH-entry ELF kernel
mini-fc restore --from DIR   resume from a snapshot directory
mini-fc version | help

run flags:

Flag	Default	Notes
--kernel PATH	required	ELF kernel with a Xen PVH note
--mem MIB	128	guest RAM
--cmdline STR	console=ttyS0 reboot=k panic=1 nomodules	virtio_mmio.device= fragments are auto-appended per device
--drive path=PATH[,ro]	none, repeatable	first → /dev/vda, second → /dev/vdb, …
--net tap=NAME[,mac=AA:…]	none, repeatable	first → eth0, …
--snapshot-after DURATION	0	dump state after this wallclock duration
--snapshot-to DIR	required if --snapshot-after	snapshot directory
--trace	false	log every KVM exit to stderr

Architecture

One process per VM. Each VMM owns:

• one /dev/kvm fd, one VM fd, one in-kernel IRQchip + 8254 PIT
• one anonymous-mmap guest memory region (KVM slot 0)
• one vCPU pinned to its OS thread (KVM is per-thread)
• one minimal 16550A UART stub at port 0x3f8 for the serial console
• zero or more virtio-MMIO devices, each in a 4 KiB slot starting at 0xd0000000 with consecutive ISA IRQs from 5

The hot path is a single-threaded KVM_RUN loop in pkg/vmm/vmm.go that dispatches KVM_EXIT_IO to the serial stub, KVM_EXIT_MMIO to the matching virtio transport, and KVM_EXIT_HLT / KVM_EXIT_SHUTDOWN to teardown. Async work — tap RX, the snapshot timer — runs in goroutines that pulse the in-kernel IRQ chip via KVM_IRQ_LINE.

For the longer story (mapping to upstream Firecracker, boot protocol details, virtio mechanics) see docs/architecture.md and the ADRs in docs/adrs/.

Layout

mini-firecracker/
├── cmd/mini-fc/         CLI: check / run / restore
├── pkg/kvm/             /dev/kvm ioctl wrappers, struct mirrors,
│                        vCPU/VM state save & restore
├── pkg/boot/            ELF + PVH note + hvm_start_info + initial regs
├── pkg/virtio/          virtio-MMIO transport, split virtqueue,
│                        block, net
├── pkg/tap/             /dev/net/tun wrapper
├── pkg/vmm/             KVM_RUN loop, MMIO/IO dispatch, 16550A serial,
│                        snapshot pause + restore orchestration
├── pkg/snapshot/        on-disk format (manifest + state.json + memory.bin)
├── internal/hostcheck/  implementation of `mini-fc check`
├── docs/                architecture, boot-protocol notes, ADRs
└── testdata/            kernel + rootfs (gitignored — see
                         testdata/README.md for download URLs)

Status & known limits

Tested end-to-end on the firecracker-ci 5.10 vmlinux + Ubuntu 24.04 squashfs — the same artifacts upstream Firecracker ships for integration tests.

Known gaps, in rough priority order:

• Linux 6.1 firecracker-ci kernel needs ACPI for virtio discovery — its cmdline-based path (virtio_mmio.device=…) is stripped at build time. mini-fc would need minimal ACPI table generation to support it.
• Snapshot-while-idle doesn't fire — the timer is checked at the next KVM_RUN boundary, and a HLT'd guest stays in-kernel until IRQ wake. KVM_SET_SIGNAL_MASK + tgkill would preempt it.
• Restore copies memory from disk into a fresh anonymous mmap. mmap(MAP_PRIVATE, fd) of memory.bin would skip the copy.
• CPUID is re-derived on restore rather than saved. Fine on the same host; cross-host migration would need it explicit.

Non-goals (deliberate, not bugs): no jailer, no REST API, no PCI, no GPU, no Windows guests, no live migration. mini-firecracker stays small enough to read in an afternoon.

Why this exists

Firecracker is the production-VM-shaped sibling of gVisor — same multi-tenant-FaaS use cases, opposite trade-offs (full kernel vs shrunk kernel, full Linux compat vs ~125 ms cold start). Reading the upstream Rust works, but building a small version makes it click. mini-firecracker is that small version: one phase per concern, one package per layer, upstream parallels named in code comments.

Go (instead of matching upstream Rust) keeps the dependency surface to a single import (golang.org/x/sys) and lets the mini-sentry ↔ mini-firecracker cross-reference read cleanly.

Reading further

• KVM API: https://docs.kernel.org/virt/kvm/api.html
• virtio v1.2 spec: https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html
• Xen PVH boot protocol: https://xenbits.xen.org/docs/unstable/misc/pvh.html
• Upstream Firecracker: https://github.com/firecracker-microvm/firecracker
• Sibling project: https://github.com/mtclinton/mini-sentry