[Bug]: `sandbox exec` hangs after receiving complete gRPC response — never outputs data, never exits

[nv-ddave]

Agent Diagnostic

Investigated using openshell sandbox exec -vvv trace on a DGX Spark host (aarch64). The gRPC exec flow completes server-side, but the CLI never prints stdout and never terminates.

Findings from -vvv trace:

• TLS handshake to local gateway (127.0.0.1:8080) succeeds, h2 connection established.
• The ExecSandbox gRPC request is sent on stream StreamId(1).
• Server responds with HTTP 200 headers (Headers { stream_id: StreamId(1), flags: END_HEADERS }).
• 1,780 bytes of command output are received in a Data frame on StreamId(1).
• Server sends trailing headers with END_HEADERS | END_STREAM — the stream transitions to Closed(EndStream).
• At this point the gRPC call is fully complete.
• The CLI never writes the received data to stdout and never exits. Instead, the h2 connection stays Open and enters a keepalive loop, exchanging PING frames every ~10 seconds indefinitely.
• Without an external timeout, the process hangs until the SSH session times out (exit code 255).

Likely root cause: The client-side exec output streaming code receives the ExecSandboxEvent data frames but does not flush them to stdout or signal process exit after the server closes the stream. The h2 connection teardown / process exit path is not triggered by the stream completing.

Description

openshell sandbox exec hangs indefinitely after the remote command completes. The gRPC response (stdout data + trailing headers with END_STREAM) is received in full, but the CLI never prints output and never exits.

This affects both --tty and --no-tty modes. The --timeout flag on exec does not help (tested with --timeout 15); only an external timeout command or SSH session timeout terminates the process.

The ssh-config + SSH workaround (openshell sandbox ssh-config → ssh -T -F) works correctly for the same commands.

Reproduction Steps

# 1. Verify sandbox is ready
openshell sandbox list
# NAME          NAMESPACE  CREATED              PHASE
# my-assistant  openshell  2026-03-22 08:31:11  Ready

# 2. Try exec — hangs forever, no output
openshell sandbox exec -n my-assistant --no-tty -- echo "hello"
# (hangs — no output, never exits)

# 3. Same with --tty
openshell sandbox exec -n my-assistant --tty -- echo "hello"
# (hangs — no output, never exits)

# 4. With --timeout flag (still hangs — timeout flag has no effect)
openshell sandbox exec -n my-assistant --no-tty --timeout 15 -- echo "hello"
# (hangs for 15+ seconds, no output, does not exit)

# 5. Wrapped in external timeout — exits with 124 (killed), still no output
timeout 30 openshell sandbox exec -vvv --no-tty --timeout 15 -n my-assistant -- echo "test"
# (verbose trace shows 1780 bytes received, stream closed, then PING loop until kill)

# 6. SSH workaround works fine
openshell sandbox ssh-config my-assistant > /tmp/sc
ssh -T -F /tmp/sc openshell-my-assistant "echo hello"
# hello   ← works instantly

Expected Behavior

openshell sandbox exec -n my-assistant -- echo "hello" should print hello to stdout and exit with code 0, similar to docker exec or kubectl exec.

Actual Behavior

The CLI receives the complete gRPC response (verified via -vvv trace showing 1,780 bytes of data and END_STREAM) but never prints output and never exits. The h2 connection remains open, sending PING keepalive frames every 10 seconds indefinitely.

Environment

• OpenShell version: openshell m-dev (installed via install.sh, which resolved to v0.0.28)
• Host OS: Ubuntu (NVIDIA DGX Spark), Linux 6.17.0-1008-nvidia aarch64
• Host architecture: aarch64
• Sandbox image: Default NemoClaw sandbox (created 2026-03-22)
• Gateway: Local (127.0.0.1:8080)
• Install method: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

Abbreviated -vvv Trace (Key Moments)

# Connection established
2026-04-14T13:55:04 DEBUG connecting to 127.0.0.1:8080
2026-04-14T13:55:04 DEBUG connected to 127.0.0.1:8080
2026-04-14T13:55:04 DEBUG Using ciphersuite TLS13_AES_256_GCM_SHA384

# Request sent, response headers received
2026-04-14T13:55:04 DEBUG send frame=Headers { stream_id: StreamId(1), flags: END_HEADERS }
2026-04-14T13:55:04 DEBUG received frame=Headers { stream_id: StreamId(1), flags: END_HEADERS }

# 1780 bytes of command output received
2026-04-14T13:55:04 DEBUG received frame=Data { stream_id: StreamId(1) }
2026-04-14T13:55:04 TRACE recv_data; size=1780; connection=5242880; stream=2097152

# Stream fully closed by server
2026-04-14T13:55:04 DEBUG received frame=Headers { stream_id: StreamId(1), flags: END_HEADERS | END_STREAM }
2026-04-14T13:55:04 TRACE recv_close: HalfClosedLocal => Closed
2026-04-14T13:55:04 TRACE transition_after; stream=StreamId(1); state=Closed(EndStream)

# CLI does NOT exit — enters PING keepalive loop
2026-04-14T13:55:14 DEBUG send frame=Ping { ack: false, payload: [...] }
2026-04-14T13:55:14 DEBUG received frame=Ping { ack: true, payload: [...] }
2026-04-14T13:55:24 DEBUG send frame=Ping { ack: false, payload: [...] }
2026-04-14T13:55:24 DEBUG received frame=Ping { ack: true, payload: [...] }
# ... repeats every 10s until external kill

Impact

• sandbox exec is unusable for scripting, CI/CD, and agent tool-use (its primary purpose per feat(cli): add openshell sandbox exec subcommand #750)
• Workaround exists: openshell sandbox ssh-config + ssh -T -F (used in production by NemoClaw bridges)
• The --timeout flag on exec also appears non-functional (does not terminate the command)

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why (client-side CLI bug in gRPC response handling)