fix(codex): inject inbound immediately — remove the idle gate (ADR-060) - #294
Conversation
Codex inbound queued every message and injected only during a confirmed-idle window. That gate rested on an assumption ADR-057 recorded as "untested at the time of writing": that a mid-turn turn/start would interleave and corrupt the thread. It has now been measured, and it is false. Two consequences followed from it. It duplicated a guarantee Codex already makes — its followup_task contract delivers at a message boundary while sampling, or after the pending tool call completes. And it deadlocked: a thread blocked in collaboration.wait_agent reports `active`, so we queued rather than injected, withholding the very message that would have released the agent. Delivery is now immediate, into a busy thread. Deleted waitForIdle, idleDeadlineMs, queueWaitWarnMs, noteLongWait, the long-wait flag, QueuedInbound.queuedAt and fmtDuration. Net deletion. KEPT, deliberately: a best-effort skip while the last observed status is "compacting" (untested conservatism, not a measured requirement); the queue as a retry buffer for genuine TRANSPORT failures with all of #287's logging; and thread/read for dashboard status only, never as a delivery gate. The compacting skip is bounded and self-correcting, because review caught that the naive version reintroduced the #287 class through a new door: lastStatus is a CACHE, and nothing reset it. A missed "compaction finished" push — we are a non-creator, so it is never replayed — combined with a failing thread/read would pin it forever, queueing every message with no operator notification, while the only warning said the DASHBOARD looked stale. Now cleared on teardown, refreshed once when unknown, bounded at 5 minutes, and it fails OPEN with a notification saying we stopped believing the status. Also folds in the last true silent drop: shutdownAllAttachments calls disposeCodexControl on the shutdown PATH, so a restart mid-queue logs the buffered inbound instead of discarding it in silence. The durable half is the log — dispose also notifies, but that lands in an in-memory store the process destroys microseconds later, which the ADR records rather than papers over. And a log that named a transport failure while naming nothing: an undici ErrorEvent carries message "", which ?? does not rescue. The invariant-level integration suite from #292 held across the removal with not one assertion moved — the only edit was dropping idlePollMs from its timings call. That is the regression evidence. Three unit tests asserting the gate's mechanics were deleted rather than rewritten; four added, including a stale-compacting test verified red against an unbounded hold. Live-QA'd against real codex 0.144.6: an agent blocked in wait_agent released ~4s after injection; an 8-step task injected at step 3 (daemon independently reporting status=active) completed all 8 files byte-correct AND answered the injected message; rollout parsed clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ArHoM9BMv2HTx2kG788VEQ
nox-0x
left a comment
There was a problem hiding this comment.
Approving — the gate removal is well-evidenced, the deletion is genuinely net-negative, and the compacting skip is bounded in the right direction; the three findings below are follow-ups, not blockers.
What I verified
deliverToCodex→ enqueue → drain →turn/startwith no idle wait; the queue's remaining consumers are all transport-failure paths (ensureThreadnull,turn/startreject, drain catch), each of which retainsnoteFailure+scheduleRetryand never drops.- The compacting guard fails open on
lastStatus === null, and the single refresh is correctly scoped to "the guard would otherwise be blind" rather than reintroducing a per-delivery read. - The
turn/start-reply shift ordering is untouched, with the #287 rationale intact in the comment. - The
ErrorEventfix is right:||over??is exactly what an empty-stringmessageneeds, and.error?.messageis where undici puts the cause. shutdownAllAttachmentsdisposes beforelive.clear(), in a synchronous loop thatpty.kill()'s asynconExitcannot preempt, and the new try/catch closes a real hole (a throw there would have skippedremovePidFile/removeControlSocket).restartAllAttachmentshas its own kill loop and is unaffected, so no controller is disposed out from under a respawn.- No stale references to
waitForIdle/idleDeadlineMs/queueWaitWarnMs/idlePollMsremain outside the historical ADR-057 text, which is correctly left alone. - ADR-060's UNTESTED section (compaction, byte equality, version skew 0.144.6 vs forge 0.145.0, sample size) is unusually honest about what the evidence does and does not cover, and the
sleepmethodology note is the kind of thing that saves the next person a day.
I could not execute the test suite in this environment, so the coverage assessment below is by reading.
Follow-ups (non-blocking)
- 🟡
codexControl.ts— the compacting bound is per-message, not per-episode.compactingSinceis cleared on fail-open and again on successful inject, butlastStatusis not, so under a genuinely stalecompactinga backlog of N drains at one message percompactingMaxHoldMswith N "delivering anyway" notifications. Not a drop, but it narrows the bound the PR body sells as protecting the queue. - 🟡
codex-control.test.ts— the stale-compacting test's/may be stale/assertion is also matched bystatusLoop's "the dashboard status may be stale" notification, which fires first understatusPollMs: 10+failThreadRead. The delivery assertion is the real (and genuinely red-against-unbounded) guard; the notification half currently passes for free. - 🟢
teardownSocketclearslastStatusbut notcompactingSince, so a reconnect can inherit an expired timestamp and fail open immediately on a genuinely new compaction, with a false "status likely stale" notification.
Version skew is worth a note for whoever operates this: the live QA is on 0.144.6 and forge runs 0.145.0. The ADR says so; just flagging that the first real-world signal will come from the newer daemon.
Review found the bound protected the head of the queue, not the queue. `compactingSince` was cleared on fail-open, so the next queued message re-armed a fresh hold: with `lastStatus` pinned at "compacting" (the exact scenario the bound exists for — a missed push plus a failing thread/read), a 5-deep backlog drained one message per bound and emitted five identical notifications. The decision is about the reported STATUS, so it now latches (`compactingDisbelieved`) until that status actually changes. A delivered message no longer ends an episode; leaving "compacting" does. Also: `teardownSocket` cleared `lastStatus` but not `compactingSince`, so a genuinely NEW compaction after a reconnect looked like it had already outlived the bound — immediate fail-open plus a "status likely stale" log and notification that were both false. Both halves of the guard's state now go together. And the stale-status test asserted on `/may be stale/`, a substring statusLoop's unrelated "the dashboard status may be stale" warning also contains — and that one fires within ~30ms, before the 60ms bound expires. It was satisfied whether or not the branch ran. Now matches `/Delivering anyway/`, unique to the branch. Added a backlog test for the case the per-message bound got wrong. Both new tests mutation-verified: reverting to the per-message bound reds the backlog test, and removing the fail-open notification reds both. Caught in review by nox-0x. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ArHoM9BMv2HTx2kG788VEQ
|
All three addressed in 29cb77b — the first was a real defect in the fix, thank you. Per-message bound (codexControl.ts:643). Correct, and it undercut the framing. Clearing
The test assertion (codex-control.test.ts:222). This is the sharpest one. Notably this is the same shape as your catch on #292 (an assertion satisfied by something other than the thing under test). Asserting on a substring of a message rather than on something unique to the branch is apparently my recurring way of writing a free assertion — worth me watching for directly. |
…w rejected it (#295) `CLAUDE.md` and ADR-060's Rationale still described the compacting skip as "a one-line skip" — the naive design review REJECTED during #294, because `lastStatus` is a cache: a failing `thread/read` or a missed "compaction finished" push pins it at `compacting` forever and inbound is silently swallowed (the #287 silent-drop class through a new door). The merged guard (`gateway/codexControl.ts:635-658`) is bounded, latching, and fails open — `compactingMaxHoldMs` bounds the hold at 5 minutes, `compactingDisbelieved` latches past the bound, both halves reset on reconnect and when the status leaves compacting, and the operator is notified when the bound trips. ADR-060 item (a) was corrected to match; these two prose sites were not carried along, leaving a future agent a documented invitation to "simplify" the guard back into the wedge. No code changes — the guard is correct as merged. ADR-060's decision and asymmetry argument are unchanged; only its stale parenthetical describing the shipped code is corrected, so it no longer contradicts its own item (a). Claude-Session: https://claude.ai/code/session_01RNhuVK6ZXgTiCLQ8h2y97x Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
What & why
Codex inbound queued every message and injected a
turn/startonly during a confirmed-idle window. That gate rested on an assumption ADR-057 itself recorded as "untested at the time of writing" — that a mid-turn injection would interleave and corrupt the thread.It has now been measured, and it is false. Two things followed from believing it:
followup_taskcontract delivers a queued message at a message boundary while sampling, or after the pending tool call completes. Codex owns mid-turn delivery safety; we re-implemented it from outside, with strictly less information.collaboration.wait_agentreportsactive. So the gate queued instead of injecting — withholding the very message that would have released the agent. The agent waited on a message the gate was holding because the agent was waiting.flowchart LR subgraph before["before — the gate"] A1["inbound"] --> B1["queue"] --> C1{"thread idle?"} C1 -->|"no (incl. wait_agent)"| B1 C1 -->|yes| D1["turn/start"] end subgraph after["after — ADR-060"] A2["inbound"] --> B2["queue<br/><i>retry buffer</i>"] --> D2["turn/start<br/><i>immediately</i>"] D2 -.->|transport failure| B2 endHow
deliverToCodex→ enqueue → drain →turn/startimmediately, into a busy thread. DeletedwaitForIdle,idleDeadlineMs,queueWaitWarnMs,noteLongWait, the long-wait flag,QueuedInbound.queuedAt, andfmtDuration. Net deletion.Kept, deliberately — the queue did not die:
"compacting". Untested conservatism, not a measured requirement — we determined nothing about compaction, not that it's unsafe.thread/readfor dashboard STATUS only. No longer a delivery gate.The compacting skip is bounded, because the naive version was a new silent drop
Worth calling out, because review caught it and it inverts the justification I'd written.
lastStatusis a cache, and nothing reset it. A missed "compaction finished" push — and we're a non-creator, so it is never replayed — combined with a failingthread/readwould pin it at"compacting"forever. Every message to that agent would queue indefinitely, logging every 5s, with no operator notification, because the compacting branch was the only hold-back path in the file that never callednoteFailure. The one notification the operator would get says the dashboard status may be stale — pointing at cosmetics while every message they send is swallowed.That's the #287 class through a new door. So the skip now: clears
lastStatuson teardown (it describes a socket that no longer exists), refreshes once when the status is unknown rather than injecting blind, bounds the hold at 5 minutes, and then fails open with a notification saying we stopped believing the status. Guarded by a test verified red against the unbounded version.This is the ADR's own asymmetry principle applied honestly: I justified the skip as cheap conservatism, and it wasn't cheap until it was bounded.
Also in this PR
shutdownAllAttachmentsnow callsdisposeCodexControlon the shutdown path, so a restart mid-queue logs its buffered inbound instead of discarding it in silence. Verified by a controlled before/after on two throwaway servers: identical steps, no output before,DROPPING 1 undelivered inbound message(s)after. The durable half is the log —dispose()also notifies, but that lands in an in-memory store the process destroys microseconds later, which the ADR records rather than papers over.ErrorEventcarriesmessage: "", and??doesn't rescue an empty string, socontrol socket error:printed no cause.Rejected — do not resurrect
multi_agent— removes a working capability to dodge a bug in our own gate.BASE_CONTEXT"don't usewait_agent" — trains agents off a working feature to preserve a workaround.ActiveTurnNotSteerable— not reproducible; that error never fired forturn/startin 8 attempts.thread/readis byte-identical for a thread doing work and one blocked inwait_agent.Testing
make checkexit 0;biome check(no--write) clean;tsc --buildclean.idlePollMsfrom its timings call, since that knob no longer exists. Test 2 is the nice one — it proved queue-then-inject-on-idle before and proves inject-immediately-into-a-busy-thread now, because what it actually asserts is "the message arrives."Live QA against real
codex0.144.6 on an isolated dev server:wait_agent, then messagedstatus=active)idle; rollout 74 lines, 0 invalid JSONDROPPING 1 undelivered inbound message(s)Checklist
make checkpasses locallymake hero— no dashboard UI change