Phase 2: Tier 1 native execution via v86 (offload backend)

[zacharywhitley]

Background

Design: docs/native-execution-and-parallelism.md §3 (Tier 1), §4.2, §11.
Depends on #1 (the tegmentum:py-offload WIT contract).

Run native code from packages with no wasm32-wasip2 wheel by executing them in
a native CPython inside the v86 wasmmachine, reached through the offload WIT.

Tasks

• Resident guest-side dispatcher (native CPython in the v86 Linux guest) that
  reads a task, resolves entry, runs it, and writes back outcome.
• Transport over the channel available today (wasi:cli/command + serial /
  virtiofs /workspace); design so it can move to typed v86 WIT when
  boot/device worlds are wired upstream.
• Snapshot a booted, package-loaded guest and restore per call (never
  boot-per-call).
• Implement offload.run host shim binding the host to the guest dispatcher.
• Benchmark warm-call latency to resolve the open question in design §11.

Acceptance

A native-only package call (e.g. a tiny numpy op) runs end-to-end through
offload.run against a snapshot-restored guest, with a recorded warm-call
latency number.

Notes / deps

• Sibling repo ~/git/v86. Its richer WIT (boot/device) is not yet wired in,
  so Phase 2 uses the interim wasi:cli/command path.
• Codec: msgpack/json (per Prototype Phase 1: define tegmentum:py-offload WIT + reference worker #1); Arrow deferred to Phase 3.

[zacharywhitley]

Decision: Tier 1 is interactive-capable (2026-05-23)

v86 Tier 1 is fast enough for interactive use. Implications for this issue:

• The offload API and the import hook (Phase 4: import hook / proxy for native-only packages #4) may route fine-grained calls to
  Tier 1, not only coarse batches.
• A resident, snapshot-restored worker is still required (never boot-per-call);
  batching large native work remains good practice where it helps.
• The latency benchmark task here is reframed from a go/no-go gate to
  characterization — measure warm-call overhead and the emulated-CPU
  multiplier to tune API granularity and find where batching still pays off.

Recorded in docs/native-execution-and-parallelism.md §10. No design questions
remain open.

[zacharywhitley]

Progress: Tier-1 transport core landed (a48c9c5)

The v86-agnostic plumbing is in place and green in CI (reference-worker job, 13 tests):

• protocol.py — length-prefixed frames over a byte stream; msgpack control
  plane carrying the task's opaque codec-encoded args/result.
• serve.py — the resident dispatcher (guest-side worker loop). Redirects a
  task's stdout to stderr so a stray print() can't corrupt the frame stream
  (the shared-serial-console hazard).
• client.py — host-side StreamClient + SubprocessClient (a subprocess
  stand-in for a v86 guest reached over serial/virtiofs; identical transport).
• test_transport.py — round-trip ok/raised over json+msgpack, many calls on
  one resident process, stdout isolation.

Remaining (v86-specific, needs the v86 runtime)

• Get a native CPython into a v86 guest (the busybox workspace/ has no
  Python today) and run python -m py_offload.serve as the resident loop.
• Bind the host StreamClient to the guest's serial/virtiofs channel instead
  of a subprocess pipe.
• Snapshot a booted, package-loaded guest; restore per session (never boot
  per call).
• Benchmark warm-call latency to characterize the envelope (design §11 — the
  Tier-1-is-interactive decision; tune API granularity).

[zacharywhitley]

Investigation: concrete integration path (see docs/tier1-v86-integration.md)

Studied ~/git/v86. Findings that shape #2:

• Guest is busybox-only — no glibc/musl. A normal dynamic python3 won't run;
  Tier 1 needs a self-contained x86_64-linux CPython (static musl or a
  relocatable build shipped with its libs). This is the main risk — validate first.
• One UART (ttyS0) = console. No spare stream channel.
• virtiofs mounts a host directory into the guest, so the host can read/write
  files the guest sees while it runs — with no v86 Rust changes. This makes a
  file-mailbox the lowest-friction channel.
• Snapshot/restore is env-driven and works (V86_SNAPSHOT_*, V86_RESTORE_SNAPSHOT).

Recommended path (minimal/no v86 Rust changes)

Self-contained CPython + py_offload in workspace/ → workspace/init background-
launches a resident worker speaking the framed protocol over a virtiofs mailbox →
host StreamClient gets a MailboxChannel → boot virtiofs-as-root, snapshot at
"worker ready", restore per session. COM2 serial is the fallback (needs Rust work
in the v86 repo).

Decisions needed

• B2 channel: virtiofs mailbox (recommended) vs. COM2 serial.
• B1 CPython: which self-contained build, and validate it runs under busybox.
• B3 cross-repo: the guest wiring (workspace/init) lives in ~/git/v86.

Next step (no v86 runtime needed)

Implement MailboxChannel (file-based reader/writer over a shared dir) + guest-side
serve adapter, tested with a tmpdir stand-in — same approach as SubprocessClient.

[zacharywhitley]

Progress: virtiofs file-mailbox channel landed (04a4ef2)

The Tier-1 transport is now built in its real form and green in CI (reference-worker, 16 tests):

• mailbox.py — host and a guest-resident worker exchange request/response frames
  as files in a shared directory the guest sees over virtiofs. No v86 Rust
  changes needed (virtiofs is just a host dir). Atomic temp + os.replace
  framing (no torn reads); immune to a task's stdout; one outstanding request at a
  time (matches the serial worker).
• MailboxClient (host) — same run(env, task) shape as StreamClient.
• serve_mailbox / python -m py_offload.mailbox <dir> (guest resident loop).
• test_mailbox.py — tmpdir stands in for the virtiofs mount, subprocess for the
  guest; round-trip ok/raised over both codecs + many calls on one worker.

Remaining (needs the v86 runtime; tracked in docs/tier1-v86-integration.md)

• B1: a self-contained x86_64 CPython that runs in the busybox guest (the
  main risk).
• Drop CPython + py_offload into workspace/; have workspace/init
  background-launch python -m py_offload.mailbox /run/py-offload.
• Point a host MailboxClient at the guest's virtiofs-shared dir.
• Snapshot at "worker ready"; restore per session; benchmark warm-call latency.

[zacharywhitley]

Finding: Python integrates as a wasmmachine component package (no x86 guest needed)

Studied how v86 integrates apps (ssh-client, coreutils, claude-code). They
are wasm32-wasip2 components, declared by a packages/<name>.json manifest
(schema: crates/wasmmachine-types/src/package.rs) and invoked by a host-side
executor that runs wasmtime run <component>.wasm <args> over a /run/wasm/ipc
request/response mailbox. The x86 emulator (v86.wasm) is a separate concern —
apps are not native binaries in the guest.

Our python.wasm is already that kind of artifact (wit_world: wasi:cli/command).
So I registered it the same way:

• Manifest: ~/git/v86/packages/python.json (mirrors ssh-client.json; component
  • /bin/python entrypoint + the stdlib declared as a stdlib.tar.gz filesystem
    payload).
• Drove it through the unmodified coreutils-executor.sh IPC flow:
  {argv:["python","--version"]} → {"stdout":"Python 3.14.3\n","exit_code":0,"mode":"component"}. ✅

Implication

This splits #2: Python + any wasip2-buildable extensions integrate via the
component-package pattern (proven above; this is really Tier 0 inside the
wasmmachine, low-risk, reuses our existing artifact). The x86-guest route is only
needed for genuinely unported native C extensions — a narrow fallback, not the
main path.

Remaining gap for full execution via the generic executor

python.wasm is not self-contained: --version runs bare, but -c "..." needs the
stdlib (else ModuleNotFoundError: No module named 'encodings'). The executor
mounts only /workspace and sets no env, so real code execution needs one of:
(a) extract the stdlib.tar.gz payload into the overlay at Python's prefix,
(b) per-entrypoint env support (PYTHONPATH/PYTHONHOME) in the executor, or
(c) a self-contained python.wasm with a frozen stdlib.

[zacharywhitley]

Full execution wired — Python runs as a wasmmachine component end to end

Committed in v86 a94a66c:

• packages/python.json — the python.wasm application package (now committed).
• agent-component/python-executor.sh — like coreutils-executor.sh, but
  extracts the stdlib payload and mounts it with PYTHONPATH, since CPython
  isn't self-contained.

Verified through the executor IPC flow:

• python -c "import json,math; print(json.dumps({'fact6': math.factorial(6)}))"
  → {"fact6": 720}, exit 0
• running a workspace script /workspace/hello.py → hello from wasi, exit 0

This nails use #1 — the POSIX/WASI runtime for the Python console (Python +
any wasip2-buildable packages), with no x86 guest. (Benign stderr "Could not find
platform … libraries" warnings remain; cosmetic, silenceable later by shipping a
prefix layout.)

Use #2 — not-yet-ported native packages stays the separate, harder track: a
Python component (use #1) offloads an unported native extension's calls via the
py-offload contract to a native interpreter in the x86 guest (Tier 1). That still
needs the static-CPython-in-guest work tracked above + in
docs/tier1-v86-integration.md.

[zacharywhitley]

Corrected use-#2 handoff architecture (recorded in docs 397010f)

Settled how native-package execution is handed off (design doc §6.1 +
docs/tier1-v86-integration.md):

• Separate, girder-managed v86 instance per worker — never the interface
  instance. A girder actor is its own isolated WASM instance, so a pool of
  native-worker actors = a pool of isolated v86 guests (isolation, responsiveness,
  independent lifecycle, per-actor budgets).
• Handoff = the shared-directory mailbox (filesystem), not v86's CLI. A
  resident v86 runs python -m py_offload.mailbox <dir>; the routing actor writes
  a request file and reads a response file in the virtiofs-shared dir. The actor
  imports wasi:filesystem (granted to that dir) + girder host — it does not
  import wasi:cli/command for per-call work.
• CLI is only the boot mechanism. wasi:cli/command run() is run-to-completion,
  so the resident v86 is owned outside the per-call path (host-managed, or a
  loop-actor whose run() owns its lifetime); the turn-actor only does mailbox
  I/O against it.

New blockers/tasks for use #2

• B2 — decided: virtiofs mailbox (built: reference-worker/py_offload/mailbox.py).
• B5 — girder-actor ⇄ v86 wrapper: v86 isn't a girder actor (exports
  wasi:cli/command; girder hosts only turn-actor/loop-actor). Need a wrapper
  component (WAC/jco) that exports the actor world, imports the mailbox filesystem,
  and runs the handoff.
• B6 — resident-v86 lifetime owner: host-managed instance vs. a girder
  loop-actor.

Use #1 (the POSIX/console runtime via the python component package) is unaffected
and already working.

[zacharywhitley]

B1 experiment: native CPython boot blocked by a v86 virtiofs-root hang (fa9f9ea)

Actually attempted to run native CPython in the guest:

• Staged python-build-standalone cpython-3.14.5 …-musl-install_only (dynamic;
  interp /lib/ld-musl-x86_64.so.1) + supplied ld-musl-x86_64.so.1 (musl 1.2.5).
• Default p04 kernel panics on rootfstype=virtiofs (no driver).
• The repo's virtiofs kernel mounts the root (FUSE INIT + lookup), but the
  guest then hangs — spins at one kernel EIP for ~1.4B instructions, never
  reaching /init. Likely timer-IRQ/scheduling or virtio-fs completion under
  noapic/nolapic. A v86 emulator issue, not CPython.
• initramfs cold boot works but 64 MB guest RAM < 84 MB python.

So B1 is blocked on v86 runtime behavior. Realistic paths (all further v86 work):

1. Disk-image (IDE) rootfs + storage kernel — not RAM-limited, sidesteps the
   virtiofs hang. (most promising next attempt)
2. Prune python (~30–40 MB) + raise guest RAM for initramfs.
3. Fix the v86 virtiofs-root cold-boot hang (timer/IRQ) — deep emulator work.

Details + constraint matrix in docs/tier1-v86-integration.md. Use #1 (Python as
a wasmmachine component) is unaffected and working.