Skip to content

fix(codex): inject inbound immediately — remove the idle gate (ADR-060) - #294

Merged
aterrylu merged 2 commits into
mainfrom
terry/codex-gate-removal
Jul 28, 2026
Merged

fix(codex): inject inbound immediately — remove the idle gate (ADR-060)#294
aterrylu merged 2 commits into
mainfrom
terry/codex-gate-removal

Conversation

@aterrylu

Copy link
Copy Markdown
Owner

What & why

Codex inbound queued every message and injected a turn/start only during a confirmed-idle window. That gate rested on an assumption ADR-057 itself recorded as "untested at the time of writing" — that a mid-turn injection would interleave and corrupt the thread.

It has now been measured, and it is false. Two things followed from believing it:

  1. It duplicated a guarantee Codex already makes. Codex's followup_task contract delivers a queued message at a message boundary while sampling, or after the pending tool call completes. Codex owns mid-turn delivery safety; we re-implemented it from outside, with strictly less information.
  2. It deadlocked. A thread blocked in collaboration.wait_agent reports active. So the gate queued instead of injecting — withholding the very message that would have released the agent. The agent waited on a message the gate was holding because the agent was waiting.
flowchart LR
  subgraph before["before — the gate"]
    A1["inbound"] --> B1["queue"] --> C1{"thread idle?"}
    C1 -->|"no (incl. wait_agent)"| B1
    C1 -->|yes| D1["turn/start"]
  end
  subgraph after["after — ADR-060"]
    A2["inbound"] --> B2["queue<br/><i>retry buffer</i>"] --> D2["turn/start<br/><i>immediately</i>"]
    D2 -.->|transport failure| B2
  end
Loading

How

deliverToCodex → enqueue → drain → turn/start immediately, into a busy thread. Deleted waitForIdle, idleDeadlineMs, queueWaitWarnMs, noteLongWait, the long-wait flag, QueuedInbound.queuedAt, and fmtDuration. Net deletion.

Kept, deliberately — the queue did not die:

  • A best-effort skip while the last observed status is "compacting". Untested conservatism, not a measured requirement — we determined nothing about compaction, not that it's unsafe.
  • The queue as a retry buffer for genuine TRANSPORT failures — socket down, no thread yet, a refused turn — with all of fix(codex): stop losing inbound messages in silence + kill the prompt-delivery false alarm #287's logging intact. Buffer and retry, never drop.
  • thread/read for dashboard STATUS only. No longer a delivery gate.

The compacting skip is bounded, because the naive version was a new silent drop

Worth calling out, because review caught it and it inverts the justification I'd written. lastStatus is a cache, and nothing reset it. A missed "compaction finished" push — and we're a non-creator, so it is never replayed — combined with a failing thread/read would pin it at "compacting" forever. Every message to that agent would queue indefinitely, logging every 5s, with no operator notification, because the compacting branch was the only hold-back path in the file that never called noteFailure. The one notification the operator would get says the dashboard status may be stale — pointing at cosmetics while every message they send is swallowed.

That's the #287 class through a new door. So the skip now: clears lastStatus on teardown (it describes a socket that no longer exists), refreshes once when the status is unknown rather than injecting blind, bounds the hold at 5 minutes, and then fails open with a notification saying we stopped believing the status. Guarded by a test verified red against the unbounded version.

This is the ADR's own asymmetry principle applied honestly: I justified the skip as cheap conservatism, and it wasn't cheap until it was bounded.

Also in this PR

  • The last true silent drop. shutdownAllAttachments now calls disposeCodexControl on the shutdown path, so a restart mid-queue logs its buffered inbound instead of discarding it in silence. Verified by a controlled before/after on two throwaway servers: identical steps, no output before, DROPPING 1 undelivered inbound message(s) after. The durable half is the logdispose() also notifies, but that lands in an in-memory store the process destroys microseconds later, which the ADR records rather than papers over.
  • A log that named a transport failure while naming nothing. An undici ErrorEvent carries message: "", and ?? doesn't rescue an empty string, so control socket error: printed no cause.
  • Comment corrections. Seven sites still described idle-gating as current behavior. One was actively inverted — it told the reader that a dropped status notification "can never wedge inbound delivery", which after this change is precisely the wedge the bound now prevents. That's the comment a future maintainer would use to rule out the bug.
  • fix(codex): stop losing inbound messages in silence + kill the prompt-delivery false alarm #287's unreleased changeset, which still asserted "inbound is idle-gated by design (injecting a turn mid-turn corrupts the thread)" and described two deleted timeouts. Both changesets publish into the same CHANGELOG; fixed in place before it ships.

Rejected — do not resurrect

  • Suppress multi_agent — removes a working capability to dodge a bug in our own gate.
  • A BASE_CONTEXT "don't use wait_agent" — trains agents off a working feature to preserve a workaround.
  • Try-then-fallback on ActiveTurnNotSteerablenot reproducible; that error never fired for turn/start in 8 attempts.
  • Classify busy-vs-waiting before injectingprovably impossible from a non-creator client: thread/read is byte-identical for a thread doing work and one blocked in wait_agent.

Testing

  • make check exit 0; biome check (no --write) clean; tsc --build clean.
  • The regression evidence: the invariant-level integration suite from test(codex): integration-test inbound delivery across the router→daemon seam #292 held across the removal with not one assertion moved. The only edit was dropping idlePollMs from its timings call, since that knob no longer exists. Test 2 is the nice one — it proved queue-then-inject-on-idle before and proves inject-immediately-into-a-busy-thread now, because what it actually asserts is "the message arrives."
  • Unit tests: 3 asserting the gate's mechanics deleted rather than rewritten (they described behavior that no longer exists); 4 added — immediate-inject-while-busy, compacting delay-not-drop, stale-compacting bound, and the terminated-drop test re-based on a transport failure.

Live QA against real codex 0.144.6 on an isolated dev server:

scenario result
agent blocked in wait_agent, then messaged released ~4s, zero stall lines
8-step task, injected at step 3 (daemon independently reporting status=active) all 8 files byte-correct, task completed, and the injected message answered
thread integrity after mid-turn injection returned to idle; rollout 74 lines, 0 invalid JSON
restart with 1 message queued before: silence · after: DROPPING 1 undelivered inbound message(s)

Checklist

  • make check passes locally
  • Conventional-commit title
  • Architectural decision recorded — ADR-060, reversing ADR-057's idle-gate assumption (its capability-gating decision stands)
  • make hero — no dashboard UI change

Codex inbound queued every message and injected only during a confirmed-idle
window. That gate rested on an assumption ADR-057 recorded as "untested at the
time of writing": that a mid-turn turn/start would interleave and corrupt the
thread. It has now been measured, and it is false.

Two consequences followed from it. It duplicated a guarantee Codex already
makes — its followup_task contract delivers at a message boundary while
sampling, or after the pending tool call completes. And it deadlocked: a thread
blocked in collaboration.wait_agent reports `active`, so we queued rather than
injected, withholding the very message that would have released the agent.

Delivery is now immediate, into a busy thread. Deleted waitForIdle,
idleDeadlineMs, queueWaitWarnMs, noteLongWait, the long-wait flag,
QueuedInbound.queuedAt and fmtDuration. Net deletion.

KEPT, deliberately: a best-effort skip while the last observed status is
"compacting" (untested conservatism, not a measured requirement); the queue as
a retry buffer for genuine TRANSPORT failures with all of #287's logging; and
thread/read for dashboard status only, never as a delivery gate.

The compacting skip is bounded and self-correcting, because review caught that
the naive version reintroduced the #287 class through a new door: lastStatus is
a CACHE, and nothing reset it. A missed "compaction finished" push — we are a
non-creator, so it is never replayed — combined with a failing thread/read
would pin it forever, queueing every message with no operator notification,
while the only warning said the DASHBOARD looked stale. Now cleared on
teardown, refreshed once when unknown, bounded at 5 minutes, and it fails OPEN
with a notification saying we stopped believing the status.

Also folds in the last true silent drop: shutdownAllAttachments calls
disposeCodexControl on the shutdown PATH, so a restart mid-queue logs the
buffered inbound instead of discarding it in silence. The durable half is the
log — dispose also notifies, but that lands in an in-memory store the process
destroys microseconds later, which the ADR records rather than papers over.

And a log that named a transport failure while naming nothing: an undici
ErrorEvent carries message "", which ?? does not rescue.

The invariant-level integration suite from #292 held across the removal with
not one assertion moved — the only edit was dropping idlePollMs from its
timings call. That is the regression evidence. Three unit tests asserting the
gate's mechanics were deleted rather than rewritten; four added, including a
stale-compacting test verified red against an unbounded hold.

Live-QA'd against real codex 0.144.6: an agent blocked in wait_agent released
~4s after injection; an 8-step task injected at step 3 (daemon independently
reporting status=active) completed all 8 files byte-correct AND answered the
injected message; rollout parsed clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ArHoM9BMv2HTx2kG788VEQ
Comment thread packages/server/src/gateway/codexControl.ts Outdated
Comment thread packages/server/src/gateway/codexControl.ts
Comment thread packages/server/src/__tests__/codex-control.test.ts Outdated

@nox-0x nox-0x left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — the gate removal is well-evidenced, the deletion is genuinely net-negative, and the compacting skip is bounded in the right direction; the three findings below are follow-ups, not blockers.

What I verified

  • deliverToCodex → enqueue → drain → turn/start with no idle wait; the queue's remaining consumers are all transport-failure paths (ensureThread null, turn/start reject, drain catch), each of which retains noteFailure + scheduleRetry and never drops.
  • The compacting guard fails open on lastStatus === null, and the single refresh is correctly scoped to "the guard would otherwise be blind" rather than reintroducing a per-delivery read.
  • The turn/start-reply shift ordering is untouched, with the #287 rationale intact in the comment.
  • The ErrorEvent fix is right: || over ?? is exactly what an empty-string message needs, and .error?.message is where undici puts the cause.
  • shutdownAllAttachments disposes before live.clear(), in a synchronous loop that pty.kill()'s async onExit cannot preempt, and the new try/catch closes a real hole (a throw there would have skipped removePidFile/removeControlSocket). restartAllAttachments has its own kill loop and is unaffected, so no controller is disposed out from under a respawn.
  • No stale references to waitForIdle/idleDeadlineMs/queueWaitWarnMs/idlePollMs remain outside the historical ADR-057 text, which is correctly left alone.
  • ADR-060's UNTESTED section (compaction, byte equality, version skew 0.144.6 vs forge 0.145.0, sample size) is unusually honest about what the evidence does and does not cover, and the sleep methodology note is the kind of thing that saves the next person a day.

I could not execute the test suite in this environment, so the coverage assessment below is by reading.

Follow-ups (non-blocking)

  1. 🟡 codexControl.ts — the compacting bound is per-message, not per-episode. compactingSince is cleared on fail-open and again on successful inject, but lastStatus is not, so under a genuinely stale compacting a backlog of N drains at one message per compactingMaxHoldMs with N "delivering anyway" notifications. Not a drop, but it narrows the bound the PR body sells as protecting the queue.
  2. 🟡 codex-control.test.ts — the stale-compacting test's /may be stale/ assertion is also matched by statusLoop's "the dashboard status may be stale" notification, which fires first under statusPollMs: 10 + failThreadRead. The delivery assertion is the real (and genuinely red-against-unbounded) guard; the notification half currently passes for free.
  3. 🟢 teardownSocket clears lastStatus but not compactingSince, so a reconnect can inherit an expired timestamp and fail open immediately on a genuinely new compaction, with a false "status likely stale" notification.

Version skew is worth a note for whoever operates this: the live QA is on 0.144.6 and forge runs 0.145.0. The ADR says so; just flagging that the first real-world signal will come from the newer daemon.

Review found the bound protected the head of the queue, not the queue.
`compactingSince` was cleared on fail-open, so the next queued message re-armed
a fresh hold: with `lastStatus` pinned at "compacting" (the exact scenario the
bound exists for — a missed push plus a failing thread/read), a 5-deep backlog
drained one message per bound and emitted five identical notifications.

The decision is about the reported STATUS, so it now latches
(`compactingDisbelieved`) until that status actually changes. A delivered
message no longer ends an episode; leaving "compacting" does.

Also: `teardownSocket` cleared `lastStatus` but not `compactingSince`, so a
genuinely NEW compaction after a reconnect looked like it had already outlived
the bound — immediate fail-open plus a "status likely stale" log and
notification that were both false. Both halves of the guard's state now go
together.

And the stale-status test asserted on `/may be stale/`, a substring
statusLoop's unrelated "the dashboard status may be stale" warning also
contains — and that one fires within ~30ms, before the 60ms bound expires. It
was satisfied whether or not the branch ran. Now matches `/Delivering anyway/`,
unique to the branch.

Added a backlog test for the case the per-message bound got wrong. Both new
tests mutation-verified: reverting to the per-message bound reds the backlog
test, and removing the fail-open notification reds both.

Caught in review by nox-0x.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ArHoM9BMv2HTx2kG788VEQ
@aterrylu

Copy link
Copy Markdown
Owner Author

All three addressed in 29cb77b — the first was a real defect in the fix, thank you.

Per-message bound (codexControl.ts:643). Correct, and it undercut the framing. Clearing compactingSince on fail-open re-armed a full hold for each following message, so with lastStatus pinned the queue drained one message per bound with N identical notifications. The decision is about the reported status, so it now latches (compactingDisbelieved) until that status actually changes — a delivered message no longer ends an episode; leaving compacting does. Added a backlog test, and mutation-verified it: reverting to the per-message bound reds it.

compactingSince surviving teardown (:334). Fixed — both halves of the guard's state now reset together. Your sequence is exactly right: a genuinely new compaction after reconnect would have looked like it had already outlived the bound, producing a "status likely stale" log and notification that were simply false.

The test assertion (codex-control.test.ts:222). This is the sharpest one. /may be stale/ also matches statusLoop's "the dashboard status may be stale", which fires ~30ms in — before the 60ms bound expires — so the notification half was free regardless of whether the branch ran. Now matches /Delivering anyway/. Verified by removing the fail-open notifier: both tests go red, which they did not before.

Notably this is the same shape as your catch on #292 (an assertion satisfied by something other than the thing under test). Asserting on a substring of a message rather than on something unique to the branch is apparently my recurring way of writing a free assertion — worth me watching for directly.

@aterrylu
aterrylu merged commit 59aade5 into main Jul 28, 2026
5 checks passed
@aterrylu
aterrylu deleted the terry/codex-gate-removal branch July 28, 2026 11:57
aterrylu added a commit that referenced this pull request Jul 29, 2026
…w rejected it (#295)

`CLAUDE.md` and ADR-060's Rationale still described the compacting skip as
"a one-line skip" — the naive design review REJECTED during #294, because
`lastStatus` is a cache: a failing `thread/read` or a missed "compaction
finished" push pins it at `compacting` forever and inbound is silently
swallowed (the #287 silent-drop class through a new door).

The merged guard (`gateway/codexControl.ts:635-658`) is bounded, latching,
and fails open — `compactingMaxHoldMs` bounds the hold at 5 minutes,
`compactingDisbelieved` latches past the bound, both halves reset on
reconnect and when the status leaves compacting, and the operator is
notified when the bound trips. ADR-060 item (a) was corrected to match;
these two prose sites were not carried along, leaving a future agent a
documented invitation to "simplify" the guard back into the wedge.

No code changes — the guard is correct as merged. ADR-060's decision and
asymmetry argument are unchanged; only its stale parenthetical describing
the shipped code is corrected, so it no longer contradicts its own item (a).


Claude-Session: https://claude.ai/code/session_01RNhuVK6ZXgTiCLQ8h2y97x

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants