app-server: turn never finalised after item/completed with zero outstanding requests/items (stdio, 0.144.3) — caller times out at 300s and retries the whole turn

[Marvinthebored]

What Problem This Solves

A codex app-server turn can reach a state where all work is finished but the turn is never marked complete. It then sits idle for the full 300 s completion timeout, the shared client is retired, and the caller retries the entire turn from scratch. Repeated across retries this silently multiplies cost and makes the agent appear frozen mid-tool-call to the end user.

I can prove the symptom precisely. I cannot prove which side is at fault from my logs alone, so I'm filing the same report on both openai/codex and openclaw/openclaw (companion issue: openclaw/openclaw#115445) with an explicit blame-weighting and full disclosure, rather than guessing and wasting a maintainer's time.

Environment

OpenClaw	2026.7.1 (2d2ddc4)
Codex harness plugin	@openclaw/codex 2026.7.1
codex app-server	codex-cli 0.144.3 (the copy bundled inside the plugin, not the global CLI)
Transport	codex app-server --listen stdio://
Model	gpt-5.6-sol, model_context_window: 258400
Multi-agent	multi_agent_version: "v2" (native codex subagents in use)
Host	macOS 15.7.3, x86_64, node v25.9.0

Evidence

Two log records, one millisecond apart, from subsystem agent/embedded:

{
  "threadId": "019fa876-d4ff-7fa0-8d45-fe3e773c54c0",
  "turnId": "019fa8ba-ce8d-7080-9fb9-de2e49f5e1c8",
  "idleMs": 300000,
  "timeoutMs": 300000,
  "lastActivityReason": "notification:item/completed",
  "lastNotificationMethod": "item/completed",
  "activeAppServerTurnRequests": 0,
  "activeTurnItemCount": 0
}

codex app-server turn idle timed out waiting for completion

{
  "threadId": "019fa876-d4ff-7fa0-8d45-fe3e773c54c0",
  "turnId": "019fa8ba-ce8d-7080-9fb9-de2e49f5e1c8",
  "reason": "turn_completion_idle_timeout",
  "detachedSharedClient": true,
  "closedClient": false,
  "activeSharedClientLeases": 1
}

codex app-server client retired after timed-out turn

The key point: the last thing received was item/completed, and at that moment there were zero outstanding app-server turn requests and zero active turn items. By every counter available, the work was done. The turn still never closed.

Expected vs observed

• Expected: after item/completed with no outstanding requests or items, the turn is finalised and the caller proceeds.
• Observed: the turn remains open for the full 300 s idle timeout, then the client is retired and the whole turn is retried.

Impact

Measured on a real workload (an iterate-on-tests session: npm run lint, node --test, pytest):

• A single session logged 526 model calls, many returning 45–234 output tokens against ~209 K of context each.
• Cumulative input across successive retried sessions escalated: 9.5M → 10.7M → 14.6M → 28.7M → 36.1M → 34.0M → 38.5M, because each retry re-ran the same commands against a larger accumulated transcript.
• Across 12 rollouts: 182 M input / 177 M cached (97.4 % hit) / 436 K output.

Cache absorbs most of it, but the work is genuinely redone each cycle, and to the user the agent simply appears frozen part-way through a tool sequence.

Blame weighting — full disclosure

I have only OpenClaw's side of the stdio channel. I did not capture the wire traffic, so the following is reasoning, not proof:

Possibly OpenClaw (~55%)

• The idle timeout that fires is OpenClaw's own.
• detachedSharedClient: true together with activeSharedClientLeases: 1 is OpenClaw's own bookkeeping. A lease still held while the client is detached looks like a leak on this side, and is exactly the kind of state that would stop a turn being considered finished.
• If OpenClaw is waiting for a terminal event that this codex version doesn't emit in this path, that is a compatibility assumption on OpenClaw's side.

Possibly codex (~45%)

• If the app-server is expected to emit a distinct terminal turn event after item/completed and does not in this path, the completion signal never arrives and no amount of caller-side handling fixes it.
• The failure showed up in a multi_agent_version: v2 session with native subagents active, so a subagent-lifecycle path may be involved in the parent turn never finalising.

What would settle it definitively: a capture of the JSON-RPC traffic on the stdio channel across one hang. If codex emits a completion that OpenClaw ignores → OpenClaw. If codex goes silent after item/completed → codex. I'm happy to produce that capture if a maintainer tells me which side to instrument, but it requires deliberately reproducing the hang (this install has since moved to a different harness).

Reproduction

I do not have a minimal reproduction, and I want to be straight about that. Conditions under which it occurred:

• Long agentic coding turn (5–20 minutes), heavy bash tool use (358 invocations, 50 distinct), apply_patch, native codex subagents spawned.
• Occurred repeatedly across a single afternoon, surviving /reset, gateway restart and a full cold start with a verified-empty process table.

Workaround

Stop the gateway, kill the app-server, start clean, and begin a fresh session. This clears the state but does not prevent recurrence.

Related — possibly the same subsystem

Same install, same day, likely adjacent: an orphaned codex subagent rollout produced this warning every 5–15 seconds indefinitely:

Codex native subagent transcript parent did not match monitor state

child thread     : 019fa7e0-b299…   (constant)
transcript parent: 019fa7df-21a2…   (constant)
expected parent  : 019fa7f4-7739… → 019fa831-86a3… → 019fa859-8429…   (new one per gateway generation)

One dead subagent (its rollout last written 4 hours earlier) was re-reconciled against whatever thread was currently live, forever. It survived /reset, a gateway restart, and a full cold start, because the state is on disk in codex-home/sessions/**/rollout-*.jsonl — and codex never prunes or renames rollouts (1263 files, 2.7 GB, all plain .jsonl). Renaming the orphan off the *.jsonl glob stopped it immediately, which suggests the monitor discovers these by scanning rollouts.

---

Disclosure: this report was investigated and written with AI assistance (Claude), from logs on a live production install. Session contents and configuration have been deliberately excluded; only opaque identifiers, counters and version strings are included.

[Marvinthebored]

Closing this — filed prematurely on my side. The evidence I have is entirely from the caller's side of the stdio channel, so I can't yet demonstrate whether the app-server fails to emit a terminal turn event or whether the caller mishandles one. I'd rather establish that with a wire capture before taking maintainer time here. Tracking on the caller side first: openclaw/openclaw#115445 — will reopen with the capture if it points this way. Apologies for the noise.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• Codex App Server 0.145.0: tool-enabled turns never emit turn/completed after all tool results are finalized #35810

Powered by Codex Action

[Marvinthebored]

Reopening at the reporter's request — leaving it up for cross-reference value. The caveat in the body stands: I have only the caller's side of the stdio channel, so this may prove to be a caller-side fault rather than an app-server one. Posting it anyway in case the signature (item/completed received, activeAppServerTurnRequests: 0, activeTurnItemCount: 0, turn never finalised) is confirmatory for anyone hitting something similar. Companion issue on the caller side: openclaw/openclaw#115445

[Marvinthebored]

Correction — I mis-stated the mechanism

Re-reading the codex rollout for the affected turn, the framing in the issue body is wrong and I want to correct it before anyone spends time on it.

I described this as "work finished but the turn never marked complete". It is not. The rollout ends like this:

13:01:35  response_item / function_call_output    ← tool result returned to the model
13:01:35  event_msg / token_count
          … 300 s of complete silence …
13:06:35  developer message "<turn_aborted> The previous turn was interrupted on purpose"
13:06:36  event_msg / turn_aborted   reason: "interrupted"   duration_ms: 175330

The last real activity was a tool result being handed back to the model. The turn was mid-flight, awaiting the model's next response, which never arrived. The abort at 13:06:35 is the caller's own 300 s idle timeout firing, not a codex event.

So activeAppServerTurnRequests: 0 / activeTurnItemCount: 0 meant no item currently in progress, i.e. waiting on upstream — not work complete. I read those counters as completion and built the report around it. My error.

Revised assessment

The trigger was most likely an upstream stall (model request sent, no response). Independent evidence from the same install on the same day: a different provider on the same host logged API Error: Connection closed mid-response after a 368 s streaming turn, and measured WAN throughput swung between 32 and 131 Mbps with ~84 ms jitter on a cellular link during severe weather. I'd now weight network as the trigger at roughly 70–80%.

What I still think is a genuine defect

A flaky network should degrade gracefully, and here it didn't:

1. No upstream read timeout or error propagation. The stalled model request produced no error at all — just silence — so the only backstop was a blunt 300 s idle timer on the caller side.
2. Retry granularity. When it fired, the entire turn was retried from the beginning rather than the single failed request. Across retries, cumulative input escalated 9.5M → 38.5M as the same tool sequence was re-run against a growing transcript.
3. detachedSharedClient: true with activeSharedClientLeases: 1 still looks like caller-side bookkeeping worth a look.

That reframes the ask: not "why wasn't a completed turn finalised", but "why does a stalled upstream request surface as silence rather than an error, and why is the recovery a full-turn retry?"

Apologies for the noise — better to correct it than leave a misleading report standing.