Bug: subagent-codex runs intermittently never settle - the child completes its turn and exits, but the dsh-side run hangs until a server restart

[dnjaramba]

Summary

Intermittently, a subagent_codex tool call never returns. The delegated Codex child runs its turn to completion — streamed output, file side effects written — and exits, but the dsh-side run stays Running forever (UI: "Deep diving…"). Cancellation does nothing. Only restarting the dsh server clears it, and the call is then recorded as The tool call was interrupted after it was recorded, but no result was durably recorded. Its outcome is unknown. / TOOL_OUTCOME_UNKNOWN.

This breaks any unattended flow that needs a Codex delegation's return value.

Environment

• dsh 0.1.5-rc.1 (npm exec @deepseek-ai/dsh web), profile web
• Custom agent preset enabling @deepseek-ai/dsh-subagent-codex and @deepseek-ai/dsh-subagent-claude-code (tools subagent_codex / subagent_claude_code; backgroundMode: one-shot, maxDepth: provider-managed)
• Linux x86_64, Node v26.8.1
• Bundled Codex: @openai/codex 0.153.4 (profile-local); native codex-linux-x64 app-server
• Provider permissionMode varied during observations (never default, approve-for-me, and a session-level full-access preset); the hang reproduces across all of them, with default read-only sandbox too

Frequency

Over one evening: ≥6 hung calls, ≥9 settled calls. Hangs often cluster within a server instance (a freshly restarted instance hung on its very first Codex call with zero concurrency; another instance ran 5 Codex/Claude calls back-to-back with 5/5 settling). The same short prompt (git status --short) sometimes settles in ~10 s and sometimes hangs forever.

Not concurrency-related in the hung cases: several hangs were the only running call in the whole instance.

Reproduction (nondeterministic — roughly half the time)

1. Start dsh web with a profile exposing subagent_codex.
2. Open a session and send: Use subagent_codex to run git status --short and return only its output.
3. If it settles, retry the same call — eventually one never settles.

Evidence that the hang is dsh-side (the child completes and exits)

Codex children log to ~/.codex/logs_2.sqlite (table logs; body column feedback_log_body; process_uuid = pid:<n>:…). Children spawned by dsh are identifiable by app_server.client_name="deepseek-harness" (the VS Code extension logs as "VS Code").

Query used:

SELECT substr(process_uuid,1,16) pid, datetime(ts,'unixepoch') ts, level, target,
       substr(feedback_log_body,1,120) body
FROM logs WHERE process_uuid LIKE 'pid:<N>:%' ORDER BY ts DESC LIMIT 8;

• Settled run (completed normally): pid 3763198, 2026-09-10 22:23:53 → 22:24:43 UTC. Final rows: a burst of
  codex_app_server::outgoing_message: app-server event: item/agentMessage/delta targeted_connections=1.
• Hung run: pid 3789156, 22:43:48 → 22:43:58 UTC. Final rows are identical — a burst of item/agentMessage/delta. The process was gone by 22:44:47 (no child in ps); the dsh call was still Running minutes later.
• Hung run with file side effects: pid 3796085, ~22:48 → 22:49:43 UTC. This delegation actually created a file in the workspace (.review-loop/plan.md), streamed its answer, and exited — and the dsh-side call still never settled.

So the child's lifecycle and log signature are indistinguishable between settled and hung runs: the child finishes, streams its final answer, and exits; the dsh side simply never concludes the run.

Workarounds attempted

• Concurrency ruled out: reproduced with a single in-flight call, and even as the first-ever call on a freshly restarted server.
• Permission modes: default (never, read-only), approve-for-me (workspace-write), and a session-level full-access preset — all reproduce; not permission-related.
• Process level: the child always finishes and exits quickly; nothing lingers to kill. UI Cancel/Stop does not recover the call.
• Downstream mitigation: all subagent payloads are now checkpointed to files by the caller, so a hang costs only a restart — never the work product.

Impact

• The session's turn stays stuck until the dsh server is restarted; the call is then recorded as TOOL_OUTCOME_UNKNOWN, with its result permanently lost — even when the child already performed its file edits.
• Any unattended flow that needs a Codex delegation's result (e.g., a Codex↔Claude review loop) is not viable without a restart-workaround on every hang.

Additional notes

• rc.2 does not fix this: the subagent packages are byte-identical between rc.1 and rc.2 (only package.json version bumps; rc.2 release notes list two UI changes). Not retested there in depth.
• During one hung-era run the child logged ERROR codex_core::tools::router: error=collab spawn failed: no thread with id: … (collab sub-thread vs ephemeral: true thread). The child continued fine; likely unrelated, noting it for completeness.
• subagent_claude_code calls settled in every observed run (same service, different provider) — but we did not stress it; the file-registration/teardown path difference may matter.

Possible investigation pointers (speculative)

• The settlement path in @deepseek-ai/dsh-subagent-codex (CodexAppServerWire, settleRunResult, subprocessRunHandle): what does the provider await before publishing a result — a terminal turn/completed notification, child quiescence, or the subprocess owner's "prove it is quiescent" step? The wrapper (node …/@openai/codex/bin/codex.js app-server --stdio → native codex app-server --stdio) exits promptly after streaming; a race between the terminal notification and process exit would match the evidence.
• Child processes are created as ephemeral: true threads over an app-server --stdio connection; hung runs show no lingering child, so nothing is left to clean up — the handle in dsh appears orphaned rather than the process.

Happy to provide the full sqlite log extracts (or additional fresh reproductions) if useful.

[argszero]

I can't reproduce this in your environment, but the report is precise enough to name the mechanism. I traced the settlement chain on c291e7961a and found two distinct unbounded waits, and one of them explains exactly why "the child exited" is consistent with "it never settles".

Why your evidence already rules most of this out

Your strongest observation is the pair: settled and hung runs have identical final log rows (the item/agentMessage/delta burst), and in the hung case the child was already gone from ps when you looked. That eliminates "the child never finished" and narrows it to "dsh never observed a terminal fact it was waiting for".

In this provider there are two different terminal authorities, and they are not the same thing:

1. turn authority — turn/completed, resolving CodexAppServerWire.turnCompleted (wire.ts:698);
2. process authority — SubprocessHandle.done / waitForExit().

Plain "pipe closed" is covered: wire.ts:441-444 rejects the run via onInputEnd → fail() when stdout ends without a terminal turn. What is not covered is the asymmetry between the two authorities.

The unbounded await

disposeCodexChild (subagent-codex/src/run.ts:190-217) ends with await child.waitForExit() — no signal. And run.ts:216 is a second, signal-less await child.done.

waitForExit (subprocess-local/src/spawn.ts:617-620) is:

const waitForExit = async (signal?: AbortSignal): Promise<boolean> => {
  if (rangeExitObserved) return true
  return waitWithAbort(observeRangeExit(), signal)
}

and the no-signal branch in managed-owner.ts:33-35 is if (signal === undefined) { await pending; return true } — unbounded. observeRangeExit (spawn.ts:530-542) resolves only when launch.owner.waitForExit() — the managed range becoming empty — resolves.

So the picture is: terminate() starts the SIGTERM → graceMs → SIGKILL ladder (spawn.ts:550-558), which bounds process death, but the range-exit observation is a separate promise and nothing bounds it. If that observation cannot conclude, dispose() never resolves. That is precisely "child gone, dsh call still Running".

Why foreground codex calls are the ones exposed

tool-subagent/src/index.ts:207-236 (settleForegroundRun) awaits the result and then, at line 225, await Promise.allSettled([... run.dispose()]) with no timeout. dispose() → subprocessRunHandle.dispose() (subagent/src/out-of-process.ts:245-262) → teardown() → disposeCodexChild. A foreground call is correct only if teardown is guaranteed to conclude.

The two providers differ in what teardown must conclude first:

• codex: disposeCodexChild calls wire.close() first (run.ts:194), which closes the JSON-RPC transport over the child's own stdin/stdout pipes — the very pipes whose EOF you already observed — and then terminate().
• claude_code: disposeClaudeCodeChild (subagent-claude-code/src/run.ts:270-302) closes the SDK query first, then terminate(), then the same unbounded await child.waitForExit().

Both end at the identical unbounded wait — yet yours is the first report where teardown hangs, and your own control sentence ("subagent_claude_code settled in every observed run") is what makes this discriminating rather than speculation: the difference is what each provider must observe to conclude, not the disposal code itself. Codex's teardown has to establish quiescence over the pipes it just closed, in a grace window racing a condition that is nearly "already true" — the most fragile ordering in the chain. It also matches your observation that the hang is independent of concurrency.

Practical conclusions

1. UI Cancel/Stop cannot recover it, and that is consistent: cancellation goes through the same dispose() chain. The missing ingredient is a bound on teardown/disposal, not a different cancel path.
2. Your file-checkpoint workaround is the right shape. Out-of-tree, the same effect is reachable with a tools/execute wrapper that arms a deadline on exec.signal — exactly the shape of the in-tree guard/timeout-policy plugin (swap exec.signal for a deadline()-derived signal, delegate, substitute a structured TOOL_TIMEOUT result when your own timer wins). The precise current limit: the tool interface carries timeoutMs (core/tools/src/schema.ts:500), but no subagent tool declares one (zero hits across packages/subagent/), so the in-tree enforcement is inert for subagent_codex. With a wrapper the model gets a structured error instead of hanging, and the orphaned run simply stays parked — the event loop is not blocked, so the server survives and no restart is needed.
3. For the native fix, the codebase already contains the shape it needs in the failure path: run.ts:404-406 passes AbortSignal.timeout(Math.ceil(spec.disposeGraceMs)) to waitForExit and then awaits child.done. The normal dispose path deliberately has no such bound, so I can't tell from the source alone whether that is intentional. If it isn't, the same bounded wait plus converting an unconcluded teardown into a diagnostic (rather than an indefinite await) would make settlement total without lying about the outcome — which matters here, because today the user-visible result is TOOL_OUTCOME_UNKNOWN with the work already done on disk.

Disclosure: I could not reproduce the hang, and I have not run this against a live hung instance — the above is a source trace on c291e7961a, not a confirmed reproduction. One cheap discriminator between the two authorities above: whether the TOOL_OUTCOME_UNKNOWN note appears at the same moment the call is still shown Running (process authority won) or only after the restart (result authority never fired). Also, if you can capture the app-server's last stderr lines around a hang, the teardown stage would be decidable directly.