Design: container exit observability + host-side log capture (#933, #909)

[BatmanByte]

Design: container exit observability + host-side log capture

This design addresses #933 and #909.

• Expose container exit — box state, exit code, and resource cleanup #933: a container entrypoint can exit while the box remains Running; the exit code is unavailable and the VM may stay allocated.
• Opt-in host-side capture of container stdout/stderr (container logs) #909: the host cannot consume the entrypoint's stdout/stderr, and an unread pipe can eventually block the workload.

Read this first

The two issues share one missing path: the guest can see the init process and its pipes, but the host has no durable record of either fact.

flowchart LR
    Init["container init<br/>the workload"]
    Guest["guest<br/>observe exit + drain output"]
    Shim["shim<br/>persist host-side facts"]
    Runtime["runtime<br/>reconcile SDK state"]

    Init --> Guest --> Shim --> Runtime

Loading

In one sentence: init runs, the guest observes, the shim persists, and the runtime reports.

The design keeps two concerns separate:

Concern	Meaning
Terminal state	How init ended: Exited(code) or Failed(reason)
Resource action	What to do with the VM and box: stop, keep, or remove

Runtime map

Component	Location	Responsibility
host runtime	Host, embedded in the client process	Starts boxes; persists SDK state; reconciles durable facts; eventually deletes boxes
boxlite-shim	Host, one per box	Hosts the VM; outlives the client; monitors the guest; writes files; tears down the VM
boxlite-guest	Inside the microVM	gRPC server; creates init; observes it; drains stdout/stderr
zygote	Single-threaded guest child	Creates container processes and can waitpid processes it created
container init	Inside the OCI container	The image's ENTRYPOINT + CMD, i.e. the actual workload

Two details matter throughout the design:

1. A live shim does not mean a live init. libkrun runs the VM inside the shim process, so ending the shim releases VM memory.
2. Init and exec are different. The zygote already waits for exec processes it creates, but init is currently created by the guest main process, so nobody keeps its exit result.

What is broken today

Area	Current behavior	Consequence
Init exit	Init is checked for liveness once after start; no background task reaps it	A fast exit races with startup, exit code is lost, and status can remain Running
Output	Init stdout/stderr are connected to pipes but normally not read	No host log file; roughly one pipe buffer of output can block init on write
Host state	Runtime watches the shim PID, not init	A live VM can hide a finished workload

The existing exit / exit.previous files are shim/VM crash records, not container exit records. logs/console.log is the VM serial console, not container stdout/stderr.

Target behavior

Exit and log paths

flowchart TB
    Init["init"] -->|"exit"| Supervisor["guest InitSupervisor"]
    Supervisor -->|"Container.Wait"| Monitor["shim monitor"]
    Monitor --> Outcome["state/container-exit.json"]
    Outcome --> Reconcile["runtime reconcile"]
    Reconcile --> State["Exited / Failed"]

    Init -->|"stdout / stderr"| Drain["guest's only pipe reader"]
    Drain --> Tail["bounded diagnostic tail"]
    Drain --> Manager["InitOutputBuffer"]
    Manager -->|"Container.StreamOutput"| LogTask["shim log task"]
    LogTask --> Log["logs/entrypoint.log"]

Loading

The exit path determines state. The output path determines observability. They run concurrently; neither is allowed to block the other.

End-to-end sequence

sequenceDiagram
    participant R as host runtime
    participant S as shim monitor
    participant G as guest
    participant Z as zygote
    participant I as init

    R->>S: InstanceSpec(container_id, generation)
    S->>G: Container.Wait (may arrive before init exists)
    R->>G: Container.Init(generation)
    G->>Z: build init
    Z->>I: create and start
    I-->>G: stdout / stderr
    G->>G: drain pipes and monitor init
    I-->>Z: exit
    Z-->>G: exit result
    G->>G: wait for stdout and stderr EOF
    G-->>S: terminal Wait response
    S->>S: atomically persist outcome
    R->>R: reconcile outcome into SDK state

Loading

The ordering is intentional: Container.Wait returns only after guest-local stdout and stderr reach EOF. At that point the shim knows the guest has read all final output; it only gives the network log stream a bounded finalization window.

Guest design

Build init through the zygote

Init must be built by zygote, just like exec. The process that calls youki build() becomes init's parent and can therefore obtain its exit result through the existing Wait(pid) IPC.

BuildSpec gains bundle_path, mode: Init | Tenant, and detach. The create/start split remains valid across the process boundary because youki reconstructs start state from disk paths rather than an in-memory parent handle.

This also moves init's clone3() out of the guest's multithreaded Tokio process, which is the reason zygote exists in the first place.

InitSupervisor

There is one supervisor per init generation, not one polling loop per Container.Wait request.

State	Meaning	Container.Wait behavior
Pending	The same-generation init is not ready yet	Retryable NotReady
Ready	Init started, but has not terminated	Wait for the terminal result
Exited	Init exited or was signaled	Return code and signal immediately
StartupFailed	Create/start failed permanently	Terminal error with reason

NotReady and generation mismatch are deliberately opposite:

• NotReady means this generation is still being created, so the shim retries with backoff.
• A stale generation returns FailedPrecondition, so the old monitor stops instead of retrying forever.

The supervisor is the only task polling zygote's non-blocking Wait(pid). It stores the terminal result in tokio::sync::watch. A Container.Wait handler first reads the current watch value with borrow(); only an empty value waits with changed().await. That makes reconnect idempotent: a newly created receiver can return an already-known exit code immediately.

Always drain init output

Each init stdout/stderr pipe has exactly one reader. It runs even when log capture is disabled:

stdout pipe ─┐
             ├─> drain reader ─> diagnostic tail (always)
stderr pipe ─┘                    └> InitOutputBuffer (capture_logs=true)

The current drain worker uses blocking OS threads. The supervisor and gRPC handlers are Tokio async tasks; a handler waiting with await does not monopolize an OS thread.

PR2 adds an EOF completion signal to the two drain workers. The workers notify the supervisor after read() returns EOF; the supervisor publishes the terminal watch value only after it has both the init result and both completion signals. drain_output() remains a non-blocking snapshot of the current tail for diagnostics; it must never wait for EOF.

InitOutputBuffer and StreamOutput

The first version has exactly one log consumer: the shim. It does not need a general multi-subscriber manager. InitOutputBuffer owns the recent-output ring, one bounded channel for the active shim StreamOutput, and that channel's dropped-byte count. The drain reader feeds it; it never creates another pipe reader.

When the shim connects or reconnects, the buffer snapshots the ring and installs the one active stream while holding the same state lock. The handler sends the snapshot first and then consumes the channel, so replay and live output meet without a gap. A second still-active stream is rejected; the shim must not reconnect in parallel.

The drain path never waits for a network send. It writes the ring and uses try_send on the single channel. When the channel is full, it accumulates dropped bytes; the next successful delivery first carries ExecOutput.dropped{byte_count}. This event measures that one shim channel's overflow only. Ring overwrite and reconnect gaps remain best-effort and cannot be counted precisely without a cursor protocol. A future second consumer can justify evolving this component into a general OutputManager.

Protocol changes

Change	Purpose
ContainerInitRequest.lifecycle_generation	Identity for this init generation
Container.Wait	Shim waits for the init terminal result
Container.StreamOutput	Shim reads init stdout/stderr
ContainerWaitResponse	exit_code, signal, and exited_at
ExecOutput.dropped	Structured overflow notification for the one shim log channel
InstanceSpec.container_id + generation	Lets the shim monitor identify init before the VM enters
ContainerConfig.capture_logs	Tells guest whether to expose StreamOutput

lifecycle_generation originates in persisted BoxState, increments on every start/restart, and must flow through InstanceSpec, Container.Init, Wait, StreamOutput, and container-exit.json. Runtime only trusts an outcome whose generation equals the current box generation.

Shim design

Monitor and durable outcome

The shim starts its monitor thread before krun_start_enter(), because that call takes over the main shim thread for the lifetime of the VM. The monitor retries Wait and, when requested, StreamOutput while the shim/VM identity is verified. It stops only when the VM is gone, the generation is stale, or guest returns a terminal startup failure.

On a terminal result, the shim writes one durable fact:

{"schema_version":1,"lifecycle_generation":7,"outcome":"exited","exit_code":7,"signal":null,"exited_at":"..."}

Startup failure uses outcome:"failed" with failure_kind:"startup_failed" and a reason.

The write must be crash-durable:

write temp -> fsync(temp) -> rename(temp, final) -> fsync(state_dir)

Writing to a temporary file prevents readers from seeing partial JSON. fsync(temp) persists file data; same-directory rename makes replacement atomic; fsync(state_dir) persists the directory entry change across a host crash.

The file lives in state/container-exit.json, not in the box root. state/ must exist before Jailer::prepare() and receive directory-level write permission; two pre-authorized files are insufficient because rename modifies the parent directory. If any write step fails, the shim retries before claiming a terminal state or shutting down the VM. If it ultimately cannot persist the result, it exits and recovery turns a formerly active box into Failed.

Logs and resource action

With capture_logs=true, the shim runs Wait and StreamOutput concurrently. It reassembles byte chunks into lines, timestamps them at host receipt time, and writes k8s-file records to logs/entrypoint.log. After Wait returns, it waits for StreamOutput EOF only for a bounded interval. Logging is best-effort and must not pin lifecycle completion.

exit_action	Shim action	Runtime follow-up
stop (default)	Shut down guest and exit shim, releasing VM memory	Preserve Exited or Failed
keep	Keep shim/VM available for diagnosis	Later stop() or restart tears down residual VM
remove	Persist remove_requested:true, then release VM memory	Remove DB, disks, snapshots, sockets and box directory after shim is absent

The runtime must wait for ProcessIdentity::Absent before consuming a remove intent. It must not force-kill a shim that is still performing graceful teardown.

Runtime design

BoxState gains persisted exit_code, finished_at, and lifecycle_generation; new fields use serde defaults so old DB rows remain readable.

All state readers use one reconcile_terminal_state(config, state) path: active info(), DB fallback, list_info, recovery, health checks, and wait().

1. A Configured or Running box with a valid same-generation outcome becomes Exited or Failed.
2. Stopping and existing terminal states are not overwritten by a late outcome. This preserves an explicit stop() operation over a concurrent natural exit.
3. If the outcome is missing, corrupt, or stale while shim is alive, state is unchanged.
4. If a formerly active box has no valid outcome and shim is absent, it becomes Failed; only an explicitly persisted Stopping becomes Stopped.

The transition is a compare-and-set under the state write lock, preventing concurrent info, list, recovery, and health-check paths from racing. DB I/O happens after the in-memory transition decision.

runtime.wait() returns:

WaitOutcome { status, exit_code: Option<i32>, signal: Option<String> }

It returns immediately for existing terminal states and otherwise re-runs the same reconcile with bounded backoff. This works for detached boxes and freshly started runtimes without requiring a runtime-global event bus.

keep may leave a live VM behind Exited or Failed. Teardown must therefore depend on residual shim/VM identity, not only status == Running. Restart first tears down that residual VM, clears old outcome data, increments generation, and starts a new init.

auto_remove remains an existing option for the owning runtime's explicit stop(). It is not the natural-exit policy. Automatic deletion after init exit is exclusively exit_action=remove; recovery must not delete a natural-exit outcome before users can query its exit code or read offline logs.

Validation

A guest-only prototype previously validated the central process-parenting assumption on a real macOS arm64 VM: when zygote builds init, it becomes the parent and receives the actual exit code (/bin/false returned 1); the fast-exit path no longer reports a startup error; and a long-running init remains unaffected. The prototype was reverted.

Each implementation PR must add reproducer-first tests for the behavior it owns. The end-to-end validation matrix is:

Scenario	Expected result
Detached creator exits; init exits 7; fresh runtime opens box	get_info, list_info, and wait agree on Exited(7)
Missing or corrupt outcome	No permanent Running; runtime converges to Failed when shim is absent
Stale generation after restart	Old monitor stops with FailedPrecondition; old outcome cannot alter the new generation
Init writes more than one pipe buffer, then exits 7	No deadlock; wait() returns 7; output reaches EOF
Slow or disconnected shim log stream	Init continues; the one channel gets dropped or a best-effort replay gap
Jailer and durable-write failures	Outcome is never treated as valid until write + fsync sequence succeeds
Long-running init plus concurrent exec	Supervisor polling does not starve zygote build requests

Delivery plan

flowchart TD
    P1["PR1: guest supervision + stdio safety"] --> P2["PR2: durable exit loop, default stop"]
    P2 --> P3["PR3: wait + SDK/CLI"]
    P2 --> P4["PR4: keep lifecycle"]
    P2 --> P5["PR5: remove + auto_remove"]
    P2 --> P6["PR6: logs end to end"]

Loading

PR	Scope	Merge boundary
1	Zygote-built init, continuous stdout/stderr drain and diagnostic tail, InitSupervisor, Wait, generation, and drain EOF completion	Fold the existing #982 into this PR. Do not leave an intermediate design that reads pipes without waiting for EOF.
2	Shim monitor, state/ permission and durable outcome, Exited/Failed, reconcile and default stop	Stacked with PR1; first complete #933 vertical slice
3	WaitOutcome, SDKs, REST mapping, and boxlite wait	Consumes PR2's durable state only; does not change the exit state machine
4	Public keep behavior and residual VM lifecycle	No delete behavior
5	Public remove, remove intent, absent-process gate, full deletion	Deletion only after shim is gone
6	InitOutputBuffer, StreamOutput, replay, single-channel backpressure, shim k8s-file writer, capture_logs, log_path, and boxlite logs	Fold the former 7a/7b together: with one consumer, a guest-only stream has no useful public merge boundary. Completes #909 end to end.

PR2 can be reviewed as a stacked PR on PR1. PR3, PR4, PR5, and PR6 can be developed in parallel once PR2's interface is stable. A stacked PR can be developed in parallel, but an intermediate change that exposes incomplete user semantics must not ship by itself.

References

• Kata agent: WaitProcess and stdout/stderr streaming over vsock.
• Podman conmon: monitor-owned exit file and k8s-file logging.
• Docker: persisted exit state and wait() semantics.
• containerd shim v2: shim lifecycle and IO model.