|
| 1 | +## Phase 3 Day 4 — real Carrier console socketpair (`/dev/hvc1` ↔ host Unix socket) |
| 2 | + |
| 3 | +> Outcome log. Status: complete. Bytes now flow guest↔host on |
| 4 | +> `/dev/hvc1` for Mac microVMs; first-party capsules |
| 5 | +> (`chat`, `did-provider`, …) can talk to host providers via |
| 6 | +> `RequestEnvelope` / `ResponseEnvelope` exactly the same way |
| 7 | +> the Linux flow does over crosvm's `unix-stream` carrier. |
| 8 | +
|
| 9 | +### Goal (recap) |
| 10 | + |
| 11 | +Day 3 wired `CapsuleBackend::VzVm` into the supervisor so |
| 12 | +`elastos ps` / `status` / `stop` work on Mac. The one thing |
| 13 | +capsule code inside the VM still **could not do** was talk to |
| 14 | +the host Carrier bridge: the Vz console attachment at |
| 15 | +`elastos-vz/src/ffi/console.rs::build_carrier_console_slot` was |
| 16 | +a `pipe(2)` loop that swallowed every guest write. The Day-2 |
| 17 | +host-side `<socket_dir>/<handle>-carrier.sock` listener existed |
| 18 | +but never had a connection — the guest's `/dev/hvc1` was wired |
| 19 | +to nothing real. |
| 20 | + |
| 21 | +Day 4 replaces the placeholder pipe with a real |
| 22 | +`socketpair(AF_UNIX, SOCK_STREAM)` and routes the host-side fd |
| 23 | +straight into the Carrier bridge dispatch loop. No Unix |
| 24 | +listener / accept on Mac — the host endpoint is handed to |
| 25 | +`tokio::net::UnixStream::from_std` directly. |
| 26 | + |
| 27 | +### What landed |
| 28 | + |
| 29 | +1. **`build_carrier_console_slot` now returns a `CarrierConsole`** |
| 30 | + (`elastos-vz/src/ffi/console.rs`): |
| 31 | + |
| 32 | + ```text |
| 33 | + socketpair(AF_UNIX, SOCK_STREAM) |
| 34 | + ├── host fd ──► OwnedFd (non-blocking, owned by caller) |
| 35 | + └── vz fd ──► NSFileHandle (closeOnDealloc=true) |
| 36 | + │ |
| 37 | + ▼ |
| 38 | + dup(vz_fd) ──► NSFileHandle (closeOnDealloc=true) |
| 39 | + │ |
| 40 | + ▼ |
| 41 | + VZFileHandleSerialPortAttachment.initWithFileHandleForReading_fileHandleForWriting |
| 42 | + │ |
| 43 | + ▼ |
| 44 | + VZVirtioConsolePortConfiguration[0] |
| 45 | + │ |
| 46 | + ▼ |
| 47 | + VZVirtioConsoleDeviceConfiguration ──► guest sees `/dev/hvc1` |
| 48 | + ``` |
| 49 | + |
| 50 | + The Vz-side `dup` is necessary because `NSFileHandle` has |
| 51 | + `closeOnDealloc=true` and Apple's attachment API takes two |
| 52 | + separate handles (one for reading, one for writing). With |
| 53 | + the duplicate, each handle owns one fd; the socket endpoint |
| 54 | + stays alive until both close, then the kernel reclaims it. |
| 55 | + |
| 56 | +2. **`BuiltMachine::carrier_host_fd`** (new field). Holds the |
| 57 | + host-side `OwnedFd` so `VzProvider::load_with_vm_config` |
| 58 | + can flow it through to `RunningVm` without crossing the |
| 59 | + `VZVirtualMachineConfiguration` ownership boundary. |
| 60 | + |
| 61 | +3. **`RunningVm::take_carrier_host_fd(&mut self) -> Option<OwnedFd>`** |
| 62 | + (new). The supervisor calls this exactly once, |
| 63 | + immediately after `VzProvider::take_running_vm`. Subsequent |
| 64 | + calls return `None` — the bridge owns the fd from then on. |
| 65 | + |
| 66 | +4. **`VzMachineHandle::new`** refactor: now takes destructured |
| 67 | + `vz_config` + `kernel_console_host_read` parameters instead |
| 68 | + of a whole `BuiltMachine`. This avoids holding the non-`Send` |
| 69 | + `Retained<VZVirtioConsoleDeviceConfiguration>` across the |
| 70 | + provider's `vms.write().await` insertion point. |
| 71 | + |
| 72 | +5. **`carrier_bridge.rs` refactor** (additive): |
| 73 | + |
| 74 | + | Function | Role | |
| 75 | + |---|---| |
| 76 | + | `run_carrier_bridge_loop(stream, ctx, label)` (new, internal) | Shared per-connection dispatch loop. Reads newline-delimited `RequestEnvelope`s, dispatches via `bridge_ctx`, writes `ResponseEnvelope`s back. Body is byte-identical to the Day-2 inline loop. | |
| 77 | + | `spawn_carrier_bridge(path, …)` (unchanged signature) | Binds a Unix listener, accepts one connection, hands the stream into `run_carrier_bridge_loop`. **Linux flow byte-identical** — crosvm's `--serial type=unix-stream` connects exactly as before. | |
| 78 | + | `spawn_carrier_bridge_on_stream(stream, …)` (new) | Skips bind/accept; takes a pre-connected `tokio::net::UnixStream` and hands it into `run_carrier_bridge_loop`. **Mac flow uses this** with the carrier socketpair host fd. | |
| 79 | + |
| 80 | +6. **`Supervisor::start_capsule_vm_macos`** picks up the fd |
| 81 | + immediately after `take_running_vm`, wraps it in a |
| 82 | + `tokio::net::UnixStream::from_std`, and calls |
| 83 | + `spawn_carrier_bridge_on_stream` with the same `BridgeContext` |
| 84 | + shape as Linux. The Day-2 `<socket_dir>/<handle>-carrier.sock` |
| 85 | + path is no longer bound (no listener); it's kept on |
| 86 | + `vm_config.carrier_socket_path` for parity with the Linux |
| 87 | + manifest dump and for future diagnostic surfaces. |
| 88 | + |
| 89 | +7. **Tests** (Mac-gated unless noted): |
| 90 | + |
| 91 | + | Crate | Test | Asserts | |
| 92 | + |---|---|---| |
| 93 | + | `elastos-vz` | `carrier_slot_constructs_with_named_port` (updated) | Existing port-name + array-shape checks now also confirm `host_fd >= 0`. | |
| 94 | + | `elastos-vz` | `carrier_slot_uses_real_socketpair_with_paired_endpoints` (new) | Drops the Vz-side carrier console; observes peer-closed (EOF / EPIPE) on the host-side fd. Proves the socketpair is genuinely connected — no pipe loop. | |
| 95 | + | `elastos-server` | `spawn_carrier_bridge_on_stream_handles_ping_pong_over_socketpair` (new) | Creates a socketpair, drives one half through `spawn_carrier_bridge_on_stream`, writes a `ping` from the other half, reads `pong` back. End-to-end proof that bytes flow guest-style → host bridge → host-style response, on the same dispatch loop the Linux flow uses. | |
| 96 | + |
| 97 | +8. **Docs:** this file; `PLAN.md` Phase 3 header advances to |
| 98 | + "Day 4 complete"; `MAC.md` capability matrix updates the |
| 99 | + Carrier console row from "stub" to "real socketpair, bytes |
| 100 | + flow guest↔host". |
| 101 | + |
| 102 | +### Apple-API note that shaped Day 4 |
| 103 | + |
| 104 | +`VZFileHandleSerialPortAttachment` takes a pair of |
| 105 | +`NSFileHandle`s — one for reading, one for writing — both with |
| 106 | +`closeOnDealloc=true`. The naive shape "give Vz the same fd |
| 107 | +twice" would double-close on dealloc. The naive shape "give Vz |
| 108 | +one fd for read and one for write, both pointing at the same |
| 109 | +socketpair endpoint" needs two distinct fds, so the Vz side |
| 110 | +`dup`s its endpoint before wrapping. The kernel's socket |
| 111 | +refcount keeps the endpoint alive until both duplicates close. |
| 112 | + |
| 113 | +`Retained<VZVirtualMachineConfiguration>` (and the inner |
| 114 | +`Retained<VZVirtioConsoleDeviceConfiguration>`) are not `Send` |
| 115 | +because they hold raw Objective-C pointers. Day 4 changes |
| 116 | +`VzMachineHandle::new` to take destructured pieces so the |
| 117 | +provider's `async fn load_with_vm_config` can consume the |
| 118 | +`BuiltMachine` entirely inside a non-async block before any |
| 119 | +`await` — keeping the surrounding future `Send` so callers |
| 120 | +in `tokio::spawn` continue to compile. |
| 121 | + |
| 122 | +### What is still *not* working after Day 4 |
| 123 | + |
| 124 | +- vsock from host → guest is not yet bridged (Day 5). vsock |
| 125 | + *to* host providers is unaffected because Carrier now works. |
| 126 | +- TAP networking (capsule reachable from host LAN) is rejected |
| 127 | + with a typed entitlement error (Phase 3 Day 6+; needs |
| 128 | + `com.apple.vm.networking`). |
| 129 | +- `wait_for_exit_code` still returns `0` for every terminal |
| 130 | + state. Distinguishing clean shutdown vs crash needs |
| 131 | + `VZVirtualMachineDelegate` notifications (Day 5). |
| 132 | +- The Day-2 `<socket_dir>/<handle>-carrier.sock` filesystem |
| 133 | + path is no longer bound on Mac. It's harmless (we keep the |
| 134 | + field set on `vm_config.carrier_socket_path` for log/diag |
| 135 | + parity), but if downstream tooling tries to `accept()` on it |
| 136 | + for telemetry it will block forever. Day 5 will revisit |
| 137 | + whether to drop the field entirely or keep it for diagnostics. |
| 138 | + |
| 139 | +### Linux-untouched evidence |
| 140 | + |
| 141 | +- `scripts/check-linux-untouched.sh bcf5a0a`: green. |
| 142 | +- `carrier_bridge.rs` is in `elastos-server`, which is **not** |
| 143 | + in the Linux-untouched gate's protected paths (the gate |
| 144 | + protects `elastos-crosvm`, `elastos-runtime`, `elastos-common`, |
| 145 | + `elastos-compute` per `scripts/check-linux-untouched.sh`). |
| 146 | + The refactor is additive: `spawn_carrier_bridge`'s |
| 147 | + public signature and behaviour are unchanged for the Linux |
| 148 | + flow (still binds a listener, still accepts one connection, |
| 149 | + still uses the same per-connection dispatch loop — now via |
| 150 | + the extracted `run_carrier_bridge_loop` helper). |
| 151 | +- `cargo clippy --workspace --all-targets -- -D warnings`: |
| 152 | + clean on Mac AND Linux. |
| 153 | +- 507 tests green locally on Mac (Day 3 ended at 505; Day 4 |
| 154 | + added 1 in `elastos-vz` and 1 in `elastos-server`). |
| 155 | + |
| 156 | +### Day 5 handoff |
| 157 | + |
| 158 | +The next slices, in priority order: |
| 159 | + |
| 160 | +1. **vsock host→guest bridging.** Today the supervisor |
| 161 | + advertises `provider_vsock_port` to the capsule via the |
| 162 | + kernel command line (Day 2 work), but there's no host-side |
| 163 | + `VZVirtioSocketDevice.connectToPort` plumbing for the API |
| 164 | + provider to reach the capsule. On Linux this is implicit |
| 165 | + in `crosvm`'s vsock model; on Mac it's an explicit Apple- |
| 166 | + API loop we need to wire. |
| 167 | +2. **`VZVirtualMachineDelegate` exit codes.** Replace the |
| 168 | + polling `wait_for_exit_code` with delegate-driven |
| 169 | + notifications so the supervisor can distinguish clean |
| 170 | + shutdown from crash and log the correct status. |
| 171 | +3. **First-party capsule end-to-end on Mac.** With Carrier |
| 172 | + bytes flowing as of Day 4, the `chat` capsule should now |
| 173 | + boot and talk to the host's `LocalhostProvider`. A manual |
| 174 | + smoke test belongs in Day 5's notes; an automated one |
| 175 | + needs the cached artefacts the CI runners don't yet carry. |
0 commit comments