Skip to content

fix(codex): wire the timeout wrapper into the /review and /ship diff passes - #2379

Open
aegixx wants to merge 1 commit into
garrytan:mainfrom
aegixx:fix/codex-timeout-review-ship
Open

fix(codex): wire the timeout wrapper into the /review and /ship diff passes#2379
aegixx wants to merge 1 commit into
garrytan:mainfrom
aegixx:fix/codex-timeout-review-ship

Conversation

@aegixx

@aegixx aegixx commented Jul 28, 2026

Copy link
Copy Markdown

Refs #1036.

#1056 added _gstack_codex_timeout_wrapper to bin/gstack-codex-probe and wired it into codex/SKILL.md. The /review and /ship Codex passes never adopted it. Its entire change to review/SKILL.md was appending < /dev/null, the stdin fix from #971/#1045, so those passes still run on a bare 5-minute Bash gate and the stalls #1036 describes remain reachable there.

Before this PR, on v1.60.1.0:

file wrapper uses bare 300000
codex/SKILL.md (+ .tmpl) 5 2
review/SKILL.md 0 2
scripts/resolvers/review.ts 0 5
ship/sections/adversarial.md 0 2

The cap is the cause, not the pipe

#1036 attributes the stalls to pipe deadlock and swallowed stderr. Measuring one truncated /review adversarial pass (codex-cli 0.145.0, macOS) points somewhere simpler. Its rollout log shows task_started, no completion event, 12 shell commands and 6 web searches, last event at 287s of a 300s budget, cut mid-tool-call. The same prompt under a longer cap: rc=0 at 336s, turn.completed present. It was 36 seconds short, with no stall and no deadlock.

That also explains the roughly 50% failure rate #1036 reports: cost straddles the gate, so survival tracks diff size. --enable web_search_cached is on at both sites and accounted for about 32s of it.

Changes

For the adversarial and structured-review passes in both skills:

  1. Re-source gstack-codex-probe inside each Codex block. Shell functions do not survive between Bash blocks, so the wrapper is not in scope at the call site even though /review already sources this same file for its auth probe.
  2. Wrap both calls with _gstack_codex_timeout_wrapper 540.
  3. Raise the Bash gate to 600000, deliberately above the wrapper so the wrapper fires first and a stall surfaces as a diagnosable exit 124 instead of a harness kill returning nothing.
  4. Drop the "timeout doesn't exist on macOS" instruction. The wrapper resolves gtimeout, then timeout, then runs unwrapped, so that case is already handled; the claim is what steered these sites away from the wrapper.
  5. Treat a timeout as missing coverage rather than an informational note. The old text printed "Codex timed out after 5 minutes" and continued, which reads downstream as Codex having reviewed and found nothing.

Tests

9 new assertions in test/codex-hardening.test.ts covering wrapper presence, absence of the macOS claim, and the wrapper-under-gate ordering. They fail on the unpatched tree (29 pass / 9 fail) and pass with the fix (38/38). Full bun test green; bun run gen:skill-docs --dry-run clean, and review/SKILL.md is regenerated rather than hand-edited.

Deliberately out of scope

Three more bare timeout: 300000 sites remain in scripts/resolvers/review.ts (second opinion, plan review, and one more). This PR does not raise their gates, so their error text is left alone rather than made inconsistent with a cap that did not move. Happy to fold them in if you want.

Separately, codex/SKILL.md has the inverse ordering: a 330s wrapper under a 300s Bash gate, which makes its own exit-124 branch unreachable. Left alone here since it is the one place the wrapper is already wired, but it probably wants its own fix.

Also worth documenting

A truncated pass is not lost. Every run writes ~/.codex/sessions/<YYYY>/<MM>/<DD>/rollout-*.jsonl containing each agent_message it produced, and task_started with no completion event is a reliable truncation signature. Three substantive findings were sitting in that file after the measured run had been written off as "Codex unavailable".

Env: macOS, gstack 1.60.1.0, codex-cli 0.145.0, gtimeout/timeout both GNU coreutils 9.11.

…passes

garrytan#1056 added _gstack_codex_timeout_wrapper to bin/gstack-codex-probe and wired
it into codex/SKILL.md. The /review and /ship Codex passes never adopted it and
still run on a bare 5-minute Bash gate, so the stalls and truncation reported in
garrytan#1036 remain reachable there.

Measured on codex-cli 0.145.0 / macOS: one truncated /review adversarial pass
logged task_started with no completion event, 12 shell commands and 6 web
searches, and its last event at 287s of a 300s budget, cut mid-tool-call. The
same prompt under a longer cap completed at 336s with turn.completed present. It
was 36 seconds short. The cap is the binding constraint, not a pipe deadlock,
which also explains the roughly 50% failure rate garrytan#1036 reports: cost straddles
the gate, so survival tracks diff size.

Changes, for the adversarial and structured-review passes in both skills:

- Re-source gstack-codex-probe inside each Codex block. Shell functions do not
  survive between Bash blocks, so the wrapper is not otherwise in scope at the
  call site even though /review already sources this same file for its auth
  probe.
- Wrap both calls with _gstack_codex_timeout_wrapper 540.
- Raise the Bash gate to 600000. It must sit ABOVE the wrapper so the wrapper
  fires first and a stall surfaces as a diagnosable exit 124 instead of a
  harness kill that returns nothing.
- Drop the "timeout doesn't exist on macOS" instruction. The wrapper resolves
  gtimeout, then timeout, then runs unwrapped, so that case is already handled;
  the claim is what steered these sites away from the wrapper.
- Treat a timeout as missing coverage rather than an informational note. The
  old text printed "Codex timed out after 5 minutes" and continued, which reads
  downstream as Codex having reviewed and found nothing.

Adds 9 regression assertions to test/codex-hardening.test.ts covering wrapper
presence, absence of the macOS claim, and the wrapper-under-gate ordering. They
fail on the unpatched tree (29 pass / 9 fail) and pass with the fix (38/38).

Deliberately out of scope: three more bare `timeout: 300000` sites in
scripts/resolvers/review.ts (second opinion, plan review, and one more). This
change does not raise their gates, so their error text is left alone rather
than made inconsistent with a cap that did not move.

Note for a separate look: codex/SKILL.md has the inverse ordering, a 330s
wrapper under a 300s Bash gate, which makes its own exit-124 branch unreachable.

Refs garrytan#1036
@trunk-io

trunk-io Bot commented Jul 28, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant