dev down --remove leaves a stopped container behind when stop reports a failure

[sbeardsley]

Symptom

dev down --remove can leave a stopped but not removed container behind. Observed once on the Apple runtime:

Stopping container 'vsc-apple-test-ce-aad02693679e1ca899'...
Error: Container runtime error: Failed to stop container: XPC error reply: internalError:
failed to stop container vsc-apple-test-ce-aad02693679e1ca899 (cause: "internalError:
"XPC timeout for request to com.apple.container.runtime.container-runtime-linux.
vsc-apple-test-ce-aad02693679e1ca899/com.apple.container.sandbox/stop"")

Checking the daemon immediately afterwards showed the container had in fact stopped:

ID                                    IMAGE                            STATE
vsc-apple-test-ce-aad02693679e1ca899  docker.io/library/alpine:latest  stopped

So the stop succeeded while reporting failure, and the removal never ran. Re-running dev down --remove cleans it up.

Cause

src/commands/down.rs:36-48:

for container in &containers {
    if container.state == ContainerState::Running {
        eprintln!("Stopping container '{}'...", container.name);
        runtime.stop_container(&container.id).await?;   // <-- returns early
    }

    if remove {
        eprintln!("Removing container '{}'...", container.name);
        runtime.remove_container(&container.id).await?;

The ? propagates out of the whole function, so the if remove block is never reached. Because this is a loop, a failure on one container also abandons every remaining container in the list.

Why it surfaced now

This code is unchanged since early on — but it was effectively unreachable. Before #8, check_error in crates/apple-container/src/xpc/message.rs read the daemon's error payload with get_string, while the daemon sends it as XPC data. Every daemon error deserialized to None and returned Ok(()), so stop_container could not fail. #8 fixed that, which makes this latent ? reachable for the first time.

Scope

• Apple runtime only, as far as observed. Docker teardown was verified clean in the same session (dev up → dev shell → dev down --remove, container removed, none left behind).
• Intermittent — 1 occurrence in 4 attempts, with 3 consecutive clean up/down cycles immediately afterwards.
• Not destructive and self-recoverable — re-running dev down --remove removes the stranded container.
• The stop options fix(apple): Apple Container runtime support #8 sends ({"timeoutInSeconds": 5, "signal": 15}) match Apple's own CLI defaults (container stop defaults to SIGTERM and --time 5), so the values are not the problem. The daemon's internal XPC call to the sandbox is what times out.

Suggested fix

Two independent improvements, either of which prevents the stranded container:

1. Don't let a stop failure skip the removal. Re-check container state after a failed stop; if it is no longer running, proceed to remove. Trusting observed state over the call's return value is the same approach fix(shell): exit immediately when container shell exits #1 used for dev shell — it polls inspect_exec until running == Some(false) rather than trusting the stream to close.
2. Don't let one container abandon the rest of the loop. Collect per-container errors and continue, reporting failures at the end.

(2) is worth doing regardless of this specific timeout, since it is a plain robustness gap in teardown.

[sbeardsley]

Triage: confirmed (severity: medium — self-recoverable by re-running, but leaves stale containers and abandons the rest of teardown)

Root cause: in src/commands/down.rs, runtime.stop_container(...).await? propagates the error out of the whole function, so the if remove block never runs when stop reports a failure. Because the ? sits inside the per-container loop, one failing container also abandons every remaining container and skips the Caddy cleanup at the end. The same early-? pattern exists in run_compose_down with lower impact.

Why it surfaced now: this ? was effectively unreachable before PR #8 — the Apple runtime's stop could previously never return an error. PR #8 made stop errors real, and the Apple XPC path reports failure even when the container did in fact stop.

Fix (small): on stop error, re-check the container state via inspect/list and proceed to remove if it's no longer running (mirroring the polling approach dev shell uses); and collect per-container errors and continue the loop, reporting failures at the end instead of aborting teardown.

[sbeardsley]

Fixed by PR #19 (merged): dev down --remove no longer abandons removal when stop reports a failure; re-checks container state, accumulates per-container errors, and doesn't fail when removal succeeded despite a stop failure. Hermetic regression test added.