Conversation
The pid0 guest was exiting with status code 1 instead of 0 due to
resource handle portability issues when trying to pass custom stream
resources across WASM component boundaries.
Root Cause:
- Custom wetware:streams interface returned u32 handles that were
only valid in the host's resource table
- Guest's attempt to use these handles caused traps ("unknown handle")
- WASI stream budget violations occurred when writing without respecting
check_write() limits
- Resource cleanup caused "resource has children" errors due to
complex Cap'n Proto resource graphs
Solution:
- Switched from custom stream resources to stdio-based RPC
- pid0 guest now uses wasip2::cli::stdin/stdout for RPC transport
- Host's spawn_with_rpc() creates duplex channels for guest stdio
- Fixed StreamWriter to respect check_write() budget
- Added std::mem::forget() to avoid resource cleanup errors
Changes:
- Guest (guests/pid0): Use standard WASI stdio instead of custom streams
- Host (src/cell/executor.rs): Use spawn_with_rpc() instead of
spawn_with_streams_rpc()
- Cap'n Proto schema (capnp/peer.capnp): Simplified Executor interface
with runBytes() for bytecode-only execution
- RPC server (src/rpc/mod.rs): Implements Executor and Process interfaces
- Child guest (guests/child-echo): Simple test that prints CHILD_OK
Result:
- Guest successfully establishes Cap'n Proto RPC connection
- Spawns child process via Executor.runBytes() with raw WASM bytecode
- Child prints CHILD_OK and exits
- pid0 cleans up and exits with status code 0
Technical Details:
WASM Component Model resources are strongly typed and scoped. Passing
resource handles as u32 breaks the ownership model. Using stdio for RPC
avoids this by letting the host control what the guest's stdin/stdout
connect to (duplex channels), while Cap'n Proto treats them as standard
AsyncRead + AsyncWrite transports.
Added Executor.echo() method to validate that async Cap'n Proto RPC transports work at nested levels (child → host). Changes: - Added echo(message: Text) -> (response: Text) method to Executor interface in capnp/peer.capnp - Implemented echo method in host's ExecutorImpl that echoes back messages with 'Echo: ' prefix - Modified ExecutorImpl.run_bytes() to serve RPC system on child's stdio, allowing child processes to make RPC calls back to host - Transformed child-echo from simple stdout writer to full RPC client that makes two concurrent echo calls - Added stream adapters (StreamReader/StreamWriter) to child-echo - Added logging to track RPC call flow Architecture: Host serves Executor capability to pid0 over pid0's stdio. When pid0 calls runBytes() to spawn a child, host creates a NEW RPC system on the child's stdio serving the same Executor capability. This enables nested RPC: child → (via child's stdio) → host's Executor. Validation: - Child bootstraps RPC connection over its stdio - Child sends TWO concurrent echo requests - Host receives and responds to both calls - Validates pipelined/concurrent RPC at nested level - System exits successfully with code 0 Known Issue: Child's response logging doesn't work because its stdio is being used for RPC transport, so log messages would interfere with Cap'n Proto messages. Stderr logs work (start, rpc bootstrapped, sent requests) but stdout-based responses aren't visible. Host logs confirm RPC is working correctly. Result: Successfully demonstrates that async transports work through nested process boundaries with Cap'n Proto RPC.
- Replace the custom wetware stream resources with WASI `io/streams@0.2.9` in `wit/streams.wit`, so guests receive standard `input-stream`/`output-stream` resources. - Vendor `wasi:io@0.2.9` WIT in `wit/deps/io.wit` to satisfy `bindgen!` resolution for the local WIT package. - Update host bindings in `src/cell/proc.rs` to map WASI stream resources to `wasmtime_wasi_io::streams::DynInputStream/DynOutputStream` and implement `create-connection` with real async pipes (`tokio::io::duplex`). - Replace channel-based transport and stub pollables with `AsyncReadStream`/`AsyncWriteStream`, giving true readiness signals and backpressure from Wasmtime’s WASI IO implementation. - Simplify host RPC wiring to consume the duplex halves directly in `src/cell/executor.rs` and `src/rpc/mod.rs` (no `streams::Reader/Writer`). - Rework `DataStreamHandles` to hold a duplex stream and update tests to validate full-duplex behavior using split read/write halves. - Update guest runtime to use `wasip2::io::streams` + `wasip2::io::poll`, and replace the busy-spin loop with `wasi::io::poll::poll` in `guests/guest-runtime/src/lib.rs`. - Bump `wasip2` to `1.0.2+wasi-0.2.9` in `guests/child-echo` and `guests/pid0`, and add `wasip2` to `guests/guest-runtime` to wire bindings cleanly. - Add `wasmtime-wasi-io` dependency in `Cargo.toml` to support host-side WASI stream resources. - Update `PLAN.md` to reflect the completed async transport migration and next validation steps.
Run with: ``` # Build guest shell cargo build -p shell --target wasm32-wasip2 --release # Build cli with embedded shell guest touch src/cli/client.rs cargo build --bin ww-cli # Build node cargo run --bin ww -- run # On a separate process cargo run --bin ww-cli -- /ip4/127.0.0.1/tcp/2021 ```
Vendor stem.capnp schema, add membrane.capnp (WetwareSession with Host + Executor), and implement WetwareMembraneServer that issues epoch-scoped sessions via graft(). EpochGuardedHost and EpochGuardedExecutor wrap existing capability impls with epoch validity checks. build_membrane_rpc() bootstraps Membrane instead of bare Host. Additive change — build_peer_rpc() is untouched and still works.
Instead of regenerating stem_capnp in rs (which produces a distinct trait identity), use capnpc::crate_provides to reference stem's generated types. This lets rs use stem's MembraneServer directly rather than reimplementing it. Removes ~100 lines: WetwareMembraneServer, StatusPollerImpl, fill_epoch() — stem owns these. rs now only provides the SessionExtensionBuilder impl (the actual specialization point).
This reverts commit 28d517e.
pid0 now bootstraps via Membrane(WetwareSession) instead of a bare Host capability. On graft(), it receives an epoch-scoped Session containing Host + Executor through the WetwareSession extension. Host side: spawn_with_streams_rpc() creates a static epoch channel and uses build_membrane_rpc() instead of build_peer_rpc(). Guest side: build.rs compiles all three schemas (peer, membrane, stem) with src_prefix for clean output paths. lib.rs switches the bootstrap type and calls graft() to obtain the session.
- Guest build.rs: use src_prefix + canonicalize so capnpc outputs to $OUT_DIR/peer_capnp.rs instead of embedding absolute paths - Guest lib.rs: fix include! to use portable OUT_DIR-relative path - client.rs: remove include_bytes! for shell.wasm, make --wasm a required positional arg (reads bytes from disk at runtime) - .gitignore: add wit/cell.wit (stale local file that breaks WIT package resolution)
24 new tests covering the core host-side code paths: RPC (6): full Cap'n Proto round-trip over in-memory duplex — Host.id, Host.addrs, Host.peers, Executor.echo (basic, empty message, 5 concurrent requests) NetworkState (5): snapshot, add/remove/dedup listen addrs, set_known_peers, set_local_peer_id, clone-shares-state Loaders (7): HostPathLoader read/missing/directory, ChainLoader first-wins/fallthrough/all-fail/empty-chain ProcBuilder (6): missing bytecode/stdin/stdout/stderr errors, take_host_split once-semantics, Default trait
Fold the standalone ww-cli binary into the main ww binary as a subcommand. `ww exec <addr> <target>` now accepts either a .wasm file or an image directory containing bin/main.wasm.
Rewrite Makefile to build guests individually (child-echo before pid0 due to include_bytes! dependency) and assemble FHS-style image dirs under examples/images/. Each image has bin/main.wasm as entrypoint, consumable by `ww exec`.
The type represents a cell (orchestrates WASM process execution), not a command. Aligns the type name with the module name.
Workspace members share the root target/ by default, but pid0's include_bytes! expects child-echo's wasm in guests/child-echo/target/. Pass --target-dir target so each guest gets its own output directory.
- Move [profile.release] from child-echo/shell into workspace root (Cargo ignores profiles in non-root packages) - Strip +wasi-0.2.9 semver metadata from wasip2 version requirements in all four guest crates
Replace the two-command daemon+client pattern (ww run / ww exec) with a single ww run <image> that boots pid0 directly from an image path. - Remove Exec subcommand and all supporting code (parse_tcp_multiaddr, pump_stdin_to_bytestream, pump_bytestream_to_stdout, etc.) - Delete src/rpc/server.rs (TCP accept loop on port 2021) - Remove dead ipfs/port fields from Cell and CellBuilder - Fix FHS path: <image>/main.wasm → <image>/bin/main.wasm - Remove legacy spawn_with_rpc_internal method - Wire CellBuilder with ChainLoader (IPFS + host FS) in CLI - Add doc comments to Cell, CellBuilder, and CLI help text
4 tasks
This was referenced Feb 14, 2026
The old README documented flags (--volume, --ipfs, --preset) and workflows (two-terminal daemon+client) that no longer exist. Replace with current reality: single ww run <image> command, FHS image layout, architecture diagram, and build instructions for the staged guest pipeline.
Add boot/ to the FHS image layout spec. Each file is named with a base58btc-encoded libp2p peer ID and contains that peer's multiaddrs, one per line. The runtime will decode filenames via PeerId::from_str() and dial the listed addresses. Not yet wired into the runtime — specced in README, CellBuilder docs, and CLI help for now.
Each entry under svc/ is a nested image with its own FHS layout, spawned automatically at boot. The directory name is the service name. Services are images all the way down — they can carry their own boot/, etc/, and even nested svc/.
Captures the architectural insights from the ww-run-image work: capability-based security with no ambient authority, the runtime/pid0/children layer model, bidirectional capability flow (host -> guest -> network), the Membrane export pattern, and two-layer configuration (image config vs node config).
One config model: FHS. The image root pid0 sees is assembled from stacked layers (Stem base + positional overlays) via per-file union. Later layers override earlier; no deletes. Only the union must contain bin/main.wasm.
Move generic membrane primitives (Epoch, EpochGuard, MembraneServer, SessionExtensionBuilder) from wetware/membrane into rs as a workspace crate. stem.capnp is now canonical in rs/capnp/. This consolidates the runtime workspace: membrane is consumed as a path dep by other rs crates, and as a git dep by stem/atom.
Contributor
|
FYI — I've created a |
Add membrane crate to workspace
Wire pid0 into ww run <image>, remove exec + TCP RPC
Build fixes, tests, and dev ergonomics for pid0
Switch pid0 from bare Host to Membrane bootstrap
Add Membrane RPC bootstrap with epoch-scoped capabilities
Member
Author
|
@lthibault everything's merge, feel free to branch off master again 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.