Feature/microvm sandbox using libkrun

[allen-munsch]

this one is a bit involved but i wanted to get the idea out here it takes some of my learnings from zypi, kvm-shim, brood-box, go-microvm, and applies them to dirge

a few issues to iron out, and some force pushes to clean up my messy rebasing today

but the idea overall is taped together decently, its a step up in isolation compared to bwrap which suffers from the same issues linux cgroups do, in that its monolithic and shared kernel so a container escape in bwrap would place me on a host system

wrapping the sandbox into a lightweight vm like this with virtiofs theoretically would add a harder boundary around bash/sh stuff mitigating the need to have a million+ regex gvisor like things

i'm not sure about soft linking, symlinking between virtio-fs.

the other thing with this is that the read/write pathways are not sandboxed so they still read off host system same as !blah type commands

i tried to abstract some things out so i could attach a PTY for test harness (specifically load testing pty_harness.rs between the dirge tty over ssh to the running vm)

---

sandboxed bash runs:

example of !uname -a and dirge comparing bash things in the sandbox:

---

example of ssh into the running sandbox vm

[allen-munsch]

A concise summary of what is in this changeset for reviewers:

Core microVM backend (src/sandbox/microvm/)

• mod.rs — MicrovmSandbox with lazy-boot, SSH-based command execution, rootfs clone-per-session
• oci.rs — pure-Rust OCI image puller from Docker Hub (no buildah/skopeo needed for remote images)
• rootfs.rs — rootfs preparation from local:// (buildah) or remote OCI, layer caching by digest
• ssh.rs — ephemeral Ed25519 key generation per session, ssh2-based exec, host key injection
• runner.rs + src/bin/dirge-microvm-runner.rs — child process that calls krun_start_enter

Scheduler isolation (fixes typing stutter when VM is busy)

• renice -n 19 on the runner process — KVM vCPU threads get lowest CFS priority
• taskset -cp <last_cpu> — pins KVM away from dirge threads
• chrt --rr -p 50 — attempts to elevate dirge to SCHED_RR (needs CAP_SYS_NICE; fails silently otherwise)
• Input reader poll reduced from 5ms → 1ms

Runtime diagnostics (src/ui/input_reader.rs, gated behind sandbox-microvm feature)

• CRS-GAP, KEY-GAP, CHAN-SEND probes write to /tmp/dirge-diag.log
• tail -f /tmp/dirge-diag.log during interactive use to catch regression

PTY integration tests (src/sandbox/microvm/pty_harness.rs)

• keyboard_input_reader_load_test — 1 vCPU + guest CPU burner + 1000bps injection; p50=1.05ms p99=1.11ms
• keyboard_stress_test — 2 vCPUs + dual CPU burners + 100bps; p99 < 200ms
• Uses try_recv()+yield_now bridge (not tokio block_on) to avoid false-positive scheduling gaps

Security hardening (from review)

• --no-same-owner --no-same-permissions on tar extraction — prevents setuid-root binaries from OCI layers
• No sh -c shell interpolation — direct process piping for gzip|tar
• 2 GiB Content-Length cap on OCI blob downloads — prevents OOM from rogue registries

Docs (docs/microvm.md)

• Scheduler isolation, runtime diagnostics, PTY test reference, SSH key lifetime caveats

Two guest images: Debian (bookworm-slim) and Alpine (3.21)

---

A couple of known caveats worth calling out in the PR description:

• SSH key lifetime: ephemeral keys live in /tmp/dirge-ssh-<nanos>/ and are cleaned on MicrovmSandbox::drop. If dirge crashes or is killed, the key is gone and you cannot re-SSH into a running VM without a new session.
• buildah VFS store: stale uid/gid mappings in ~/.local/share/containers/storage/vfs/ can cause "permission denied" during buildah push. Fix is buildah rmi --storage-driver vfs <image> then rebuild.
• Workspace is read-write: commands can modify host files through /workspace — this is the same trust model as bwrap, just with a harder VM boundary around the bash process itself.

[yogthos]

Ah that's a neat idea, it would be nice to be able to do clean isolation in a micro vm and just letting the agent go yolo.

[allen-munsch]

it is so very close the main idea is there

but there's some elusive rough edges, around the agent getting hung up by read/grep/write on host vs in the sandbox, since the file paths aren't isomorphic/symmetric

$HOME vs /home/sandbox or in this case an explicit hardcoding to /workspace

the rendering refactor you did earlier was a big help in catching some, but you'll see i really went deep into the edge case testing

i need to read up on some ssh, tty, pty RFC's

the ssh session still stutters a bit and seems a little sluggish through the PTY

went as far as to consider loom, tsa+, prop testing

here's a music video of doing an attach

2026-06-05.22-20-26.mp4

---

# to setup a bigger dev image with cargo in it
# it'll place the sandbox stanza into dirge config

dirge sandbox setup --image alpine

# slim
dirge sandbox setup --image debian

# not so slim
dirge sandbox setup --image dev

[yogthos]

Reviewed the whole thing locally (worktree off the PR head). It's a serious, well-documented piece of work — pure-Rust OCI pull with digest verification, ephemeral keys, snapshotting, the scheduler-isolation tuning, and a lot of tests. But there's one merge-blocker and a handful of security/correctness gaps that matter given the feature's whole point is isolation. Grouped by severity, with file:line.

Blockers

1. The rebase reverted #388 (the v0.3.1 unified rendering/input refactor). src/ui/state.rs drops the entire InputMode state machine (input_mode field, InputMode/QuestionState/PermissionState/DialogConfirm/DialogSelect), and src/ui/mod.rs reintroduces the four nested blocking modal loops (let decision = loop at ~2289, let answer = loop at ~3280 and ~3345, let accepted = loop at ~3423). grep -c "dispatch_modal\|input_mode" ui/mod.rs ui/state.rs is 0 on this branch vs 34 on main. Merging as-is re-breaks the questionnaire-freeze bug class and undoes shipped work. Needs a clean re-rebase that preserves #388 and re-applies only the microvm-specific UI bits (pty_relay, status badge, sandbox slash command) on top.

2. src/sandbox/backend.rs (277 lines) is dead code. mod backend is never declared anywhere, so the file isn't compiled, tested, or used — the real exec logic is duplicated inline in Sandbox::exec (src/sandbox/mod.rs:340). Either wire the trait in and delete the inline duplication, or delete backend.rs. Right now it advertises an abstraction that doesn't exist.

3. No SSH host-key verification. ssh_exec (src/sandbox/microvm/ssh.rs:182) handshakes and does pubkey auth but never checks the guest host key against the injected HostKeys pubkey. The guest sshd is reached over a 127.0.0.1 ephemeral-port forward; on a shared host another local user can race/hijack that port and MITM the "sandbox" — feeding attacker-controlled output back to the agent and running the agent's commands in an attacker-controlled context. You already generate and inject a known host key, so pin it: compare session.host_key() to the injected ed25519 key before userauth.

High

4. File tools bypass the sandbox entirely. Only bash routes through the VM (src/agent/tools/bash/mod.rs:174 -> Sandbox::exec). read/write/edit/apply_patch/list_dir/find_files all hit the host filesystem directly in every mode — grep for sandbox usage under src/agent/tools/ only matches bash. So in microvm mode the agent still has full host-FS read/write through tools, which undercuts the isolation claim. At minimum call this out prominently in docs/microvm/SECURITY.md as a non-goal; ideally confine the file tools to the workspace.

5. The bash timeout is silently dropped in microVM mode. Sandbox::exec ignores timeout_secs on the SSH path (src/sandbox/mod.rs:354-406; same in the dead MicrovmBackend::exec). Only ssh.rs's 60s socket read-timeout applies, and only when the command is silent — a hung or slow-streaming guest command won't be killed at the configured timeout. Wrap the spawn_blocking in tokio::time::timeout and/or run the guest command under timeout N.

6. OCI blob size cap is bypassable. download_blob (src/sandbox/microvm/oci.rs:404) only checks Content-Length; a registry that omits it (chunked) skips the 2 GiB cap and resp.bytes() buffers the whole body in memory. Stream with a running byte counter and abort past the cap.

7. No OCI whiteout handling, and extraction leans on system tar for traversal safety. extract_or_cache_layer untars layers in sequence with no .wh./.wh..wh..opq processing (src/sandbox/microvm/oci.rs:355), so a file a later layer deletes persists — incorrect layer composition. And a malicious/typosquatted image's ../symlink members are only stopped by whatever the host tar does by default. Process whiteouts, and either extract with explicit traversal guards or document the image-trust assumption.

Medium

8. Runner stderr is discarded then "read" in dead code. mod.rs spawns the runner with .stderr(Stdio::null()) (src/sandbox/microvm/mod.rs:230) but the crash path reads child.stderr (line 287) — always None. Every assert!/expect! in dirge-microvm-runner is invisible; the user just gets "(empty)". Use Stdio::piped().

9. rootfs base-cache has no locking or atomicity. prepare() does if !cached_base.exists() { pull } (src/sandbox/microvm/rootfs.rs:44). Two concurrent sessions race into the same base dir, and a pull that fails midway leaves a partial base that later runs treat as valid. Build into a temp dir + atomic rename, guarded by a lock file.

10. Predictable temp dirs in shared /tmp. ssh.rs temp_dir() and prepare_local use temp_dir().join("...-{pid}-{nanos}") + create_dir_all, not 0700 mkdtemp (src/sandbox/microvm/ssh.rs:120, rootfs.rs:74). Predictable names in a world-writable dir invite symlink/pre-creation attacks. Use tempfile's mkdtemp.

11. Port-map bind scope unverified. The runner maps host:ssh_port -> guest:22 (src/bin/dirge-microvm-runner.rs:88). If libkrun binds 0.0.0.0 rather than 127.0.0.1, the guest sshd is network-exposed. Confirm/document localhost-only (and pin the host key per #3 regardless).

12. set_microvm_image / set_microvm_resources / ssh_connect_info use try_lock() and silently no-op on contention (backend.rs, but the same pattern would apply if wired). Switching image while the VM is busy silently does nothing — surface a busy error.

Low / polish

13. sandbox-microvm isn't in the CI matrix, so none of this (including the dead backend.rs) is built in CI. Add at least a build-only job so it doesn't bit-rot.
14. validate_snapshot_name (mod.rs:400) blocks /,\,..,empty but allows "." and control chars — tighten to an allowlist.
15. Minor TOCTOU on the ephemeral SSH port (bind -> drop -> re-bind in the runner). Acceptable, just noting.

Overall: the OCI/rootfs/ssh plumbing and the docs are strong. Please re-rebase to restore #388 first (that's the hard blocker), then the host-key pinning and the "file tools aren't sandboxed" caveat are the two I'd want resolved before this lands as a security feature.

[yogthos]

Very cool, love the video of it in action, and glad the UI refactor helped. I should have a chance to review and merge it in later today.