From 19507397c56a81e7da6bd49d804977fe8337270e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Thu, 30 Jul 2026 14:11:58 +0200 Subject: [PATCH 01/10] docs(intake): add failed-codex-call-counts-as-a-pass story --- ...ailed-codex-call-counts-as-a-pass-story.md | 140 ++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md diff --git a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md new file mode 100644 index 0000000..9d3249a --- /dev/null +++ b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md @@ -0,0 +1,140 @@ +# A failed Codex call counts as a gate pass — Story + +**Date:** 2026-07-30 · **Size:** story +**Risk:** high · **Security:** standard · **Validation:** battery+check+verification + +## 1. Problem statement + +The gate hook advances its pass counters — and, for Gate B, stores a content fingerprint +— on Codex calls that reviewed nothing. Observed directly on 2026-07-30 in this repo: +`.context/codex-gate.passCountA` moved 3 → 4 on a call that ran 272 ms and never started +a review, and 3 → 5 across two further calls that returned failures. A satisfied count +therefore overstates the passes actually held, and for Gate B the stored fingerprint +describes content nobody read, so the satisfied message reports a fresh pass covering +exactly the content that was never reviewed. That is a false ✓ in the hook's recorded +state — the direction invariant 2 names as dangerous. + +`mcp-codex-dev@1.0.1` catches its own failures and returns them as ordinary results +carrying `success: false`, so Claude Code classifies them as successful tool calls and +fires `PostToolUse`. Nothing in the hook's `PostToolUse` path inspects the result before +counting. + +Three failure shapes were captured verbatim, and one remains unproduced: + +1. **Fast fail** — the Codex result reports `success: false` with + `error.code: CODEX_EXECUTION_FAILED`. +2. **Executor timeout** — the same envelope with `error.code: CODEX_TIMEOUT`. +3. **Auto-backgrounding at 120 s** — `PostToolUse` fires mid-flight, while Codex is still + running, and the payload carries the harness's prose notice instead of any Codex + result. The eventual real completion fires no second `PostToolUse`, so the call is + counted at t=120 s and its true outcome never reaches the hook at all. This is not + hypothetical: a real Gate-B `mcp__codex__review` pass on another project hit it during + this session, at `duration_ms: 120002`. +4. **Abort during the first 120 s** — not produced; see Open questions. + +In every shape the only failure signal is double-encoded inside the tool response text; +no `isError` field exists anywhere in the payload. + +Shape 3 is preventable, and this was verified rather than assumed. With +`CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` set in the environment of the Claude Code process, +a call that runs past 120 s stays in the foreground and its `PostToolUse` payload carries +the real Codex result as parseable JSON. Measured on Claude Code 2.1.220 with two +end-to-end runs against a Codex server configured to time out at 135 s: `duration_ms` +135370 with the variable at `600000`, and 135431 with it at `0` — both past the 120 s +backgrounding point, both delivering `error.code: CODEX_TIMEOUT` as readable JSON. So `0` +disables auto-backgrounding rather than forcing it. The hook process does inherit the +variable: the same hook read `[600000]` and `[0]` on those two firings while reading +`` on firings from a session that did not set it. + +**Why this is more than a local defect.** The hook is the shipped enforcement core: every +project that installs the plugin inherits its counting semantics, so a defect here +distributes a false ✓ to every consumer rather than to this repo alone, and the failing +effect — gate state reporting reviews that never happened — undermines the workflow's +central guarantee. + +The 0.5.1 prompts already classify a timeout or abort as an incomplete pass and require +discounting it regardless of the counter. So the gap is not that no mitigation exists; it +is that the mitigation is instruction-backed and depends on the agent noticing the failed +result, while the hook's own recorded state is wrong either way and stays wrong for +whoever reads it later. + +## 2. Desired outcome + +A Codex call that did not deliver a review does not leave a pass behind. The hook's +recorded state — counters and the Gate-B fingerprint — reflects only calls whose result is +present and reports success. Where a call's outcome cannot reach the hook at all, the +operator learns that from the reminder, together with the setting that prevents it, +instead of receiving a silent ✓. The workflow stays usable on machines that lack that +setting: a discarded pass reads as an actionable setup gap, not as a failed review. + +## 3. Acceptance criteria + +- [ ] Given a `PostToolUse` payload for a Codex gate tool whose result reports failure, no + pass counter advances and no Gate-B fingerprint is stored — checked against both + payloads captured on 2026-07-30 (`CODEX_EXECUTION_FAILED`, `CODEX_TIMEOUT`). +- [ ] Given a payload carrying no readable Codex result at all — the auto-backgrounding + notice being the observed instance — no counter advances and no fingerprint is + stored. +- [ ] In that case the reminder states that the pass was discarded because the call left + the foreground, distinguishes this from a failed review, and names + `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` as the fix. +- [ ] A Codex call whose result reports success still counts exactly as it does today: the + counter advances and, for Gate B, the fingerprint is stored. +- [ ] Behaviour on all of the above is identical whether or not `jq` is on `PATH`. +- [ ] The hook exits 0 on every path above, including when the result cannot be read. +- [ ] `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` is documented where someone setting up the + workflow will see it, stating that it requires Claude Code ≥ 2.1.212 and that `0` + disables auto-backgrounding rather than forcing it. +- [ ] Regression tests cover shapes 1–3 and the success path, and the shape-1 and shape-2 + tests fail against the pre-change hook. +- [ ] The probe methodology is re-run against the changed hook and its counter readings + recorded: shapes 1–2 from the captured payloads, shape 3 with the variable absent, + shape 3 with it set, and one genuine pass that still counts. +- [ ] The plugin manifest version is bumped. + +## 4. Affected AGENTS.md invariants + +- `### Hook` — "**The hook always exits 0.** It is advisory; a reminder that can fail + closed would make the workflow unusable whenever Codex is down or the environment is + odd." Constrains the shape of "fail closed" here: withholding a *count* must not become + a non-zero *exit*, and the reminder must stay usable when the environment is odd. +- `### Hook` — "**Loose in the firing direction.** On uncertainty, fire. A missed commit + (false ✓) is the dangerous direction; a redundant warning is the accepted price." + Aligned rather than in tension: declining to count an uncertain pass leaves the gate + firing. +- `### Hook` — "**Gate-B validity is content-derived, never event-derived.** Invalidation + compares a fingerprint of the effective index plus the included worktree content, as of + the hook's invocation …" The fingerprint *store* is what must be suppressed; the + comparison semantics are unchanged. +- `### Hook` — "**POSIX `sh`, and `jq` is optional.** No bash-isms; correct behaviour via + fallback parsing when `jq` is absent." The failure signal is nested inside a JSON + string, so both parsing paths must reach it. +- `## Architecture` (Dependency direction) — "and on a Codex MCP server exposing both + `exec` and `review` (the gates key on those two tool names)." +- `### Prompts and scaffolding` — "**Prompt changes pass `docs/prompt-standards.md`** — + all 12 checklist items, for any skill, command, agent definition, hook message, or + scaffolded template." The new reminder text is a hook message. +- `### Packaging` — "**A plugin change requires a version bump.**" + +## 5. Open questions + +- Does an abort during the first 120 s fire `PostToolUse` at all, and if so with what + payload? Unproduced: `TaskStop` after backgrounding yielded no further event, so it does + not stand in for an interactive abort. If it fires nothing, aborts are not a false-✓ + source; if it fires something, its shape is unknown. +- The captured shapes are specific to `mcp-codex-dev@1.0.1`. A server mapped in via + `.context/codex-gate.tools` may report failure differently — is such a payload in scope, + or explicitly out? +- Should the absent-variable warning appear on every gate call, or once per workspace like + the existing unknown-tool note? +- `CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` exists in the Claude Code 2.1.220 binary's + environment-variable string table (2 occurrences, found by the same grep that located + the variable above), but its behaviour was never exercised and it is presumed to govern + a different surface. Treated as out of scope and deliberately kept out of the + documentation criterion, since nothing observed says what it does. Naming the wrong one + of the two in user-facing setup text would be this repo's docs-drift class. + +## 6. Suggested size + +`story` — one coherent defect with one mechanism, fitting a single spec → plan → PR; the +hook change, its tests and the setup documentation are parts of it, not separable stories. From c91a79bda875ace8ee46c22583fd91f37c8add64 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Thu, 30 Jul 2026 14:33:03 +0200 Subject: [PATCH 02/10] docs(todos): record occurrence 2 of the compound-command timing class Second consumer of the PreToolUse timing gap: is_docs_only, not tree_hash. Commit 1950739 staged exactly one docs/**.md path, yet the hook emitted the Gate-B STOP, because the commit was issued as a single Bash call whose git add had not run when the hook read the index. Direction is safe here (a false positive), unlike the tree_hash consumer. No fix; counts toward the row's trigger. Gate B skipped: behaviourally trivial (a prose occurrence note in the backlog; no executable path changes) and the change cites no profiled story, so it keeps the pre-existing judgement-based skip. Quality battery green: shellcheck clean, hook tests all passed, invariant checks 123 assertions, version-bump check 36 assertions, claude plugin validate passed. --- todos.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/todos.md b/todos.md index e16ca29..e522d9b 100644 --- a/todos.md +++ b/todos.md @@ -74,6 +74,21 @@ driven by recurrence rather than by enthusiasm. && git commit -am x` is one PreToolUse event: the hook hashes before the mutation runs, so the commit carries content the hash never saw. Consider treating any command segment preceding `git commit` as uncertain and firing. + **Occurrence 2 (2026-07-30): same event-timing class, second consumer — + `is_docs_only` rather than `tree_hash`.** Commit 1950739 staged exactly one + `docs/**.md` path, so Gate B was N/A per CLAUDE.md §5's prose exemption, yet the + hook emitted the Gate-B STOP. Cause, read from the source rather than inferred: + the docs-only branch derives its file list as + `files=$(git -C "$repo_root" diff --cached --name-only)` at PreToolUse, the commit + was issued as a single Bash call whose `git add` had not run yet, so that list was + empty — and `is_docs_only` opens with `[ -n "$1" ] || return 1`, which the branch's + own comment states as intent ("Only when the file list is POSITIVELY confirmed + docs-only; an empty list falls through to fire"). So the timing gap now has two + consumers, and this one is a **false positive** — it fires when it need not, the + safe direction under invariant 2 — where the `tree_hash` consumer above is the + dangerous direction. Counts toward this row's eventual trigger; not fixed now, and + note that any fix must keep the empty-list fallthrough firing rather than trade a + redundant warning for a missed one. - [ ] **No regression test for a `git add`/`write-tree` failure inside the throwaway index.** Derived from the code, not recalled: sections 24a-24e stub FIVE failure shapes — every checksum tool failing silently, a checksum printing a token then failing, the From 625881428aedc66cbe734204160e1e86b1f85255 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Fri, 31 Jul 2026 08:35:42 +0200 Subject: [PATCH 03/10] docs(spec): gate-pass result classification design Design for the false-positive gate-pass defect: a classifier that reads the Codex result before counting. Four classes (success / failure / backgrounded / unrecognized), envelope-keyed, failing open with disclosure on an unrecognized envelope. Records the four settled decisions with their reasons, the two anchors and their disjointness, the backgrounding anchor's residual, and what the change does not do. Also files a follow-up todos entry for the /workflow-init preflight, kept out of the story's scope deliberately. Gate B N/A: no code and no prompt paths are staged. Gate B reviews a code diff, and the routing rule forbids reviewing a document with it -- the spec's review is Gate A (mcp__codex__exec), running next per CLAUDE.md section 5. Invariant checks re-run clean against the new files. --- ...iled-codex-call-counts-as-a-pass-design.md | 218 ++++++++++++++++++ todos.md | 8 + 2 files changed, 226 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md diff --git a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md new file mode 100644 index 0000000..7f4e626 --- /dev/null +++ b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md @@ -0,0 +1,218 @@ +# Gate-pass result classification — Design + +**Date:** 2026-07-31 +**Story:** `docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md` +— read its header for the current risk/security/validation profile; this spec deliberately +does not copy those values, so there is one writable copy. + +**Evidence:** `.context/probe-payloads/` (untracked; `INDEX.md` records provenance and the +counter effect per shape). Five captured `PostToolUse` payloads, all real. + +## 1. What this changes + +The hook advances its pass counters — and, for Gate B, stores a content fingerprint — on +Codex calls that reviewed nothing, because nothing in the `PostToolUse` path inspects the +result before counting. This adds one classifier that reads the result and decides whether +the call earned a pass. + +The problem statement, the measured counter movements and the four observed payload shapes +are in the story. This spec covers only the design. + +## 2. Settled decisions + +Four questions were answered by the human before design, and their reasons are recorded +here because a later reader will otherwise re-litigate them: + +1. **Classification keys on the envelope, not the tool name.** A tool mapped via + `.context/codex-gate.tools` is in scope. +2. **An unrecognized envelope fails OPEN** — it counts, as today, plus a once-per-workspace + disclosure. Fail-closed would zero the counters permanently for every third-party + server with no fix available to the user, which is the hook's own worst failure mode + ("reviews run, counters stay 0, and the STOP fires on every commit forever — which + trains the user to ignore the hook", `codex-gate.sh`). Fail-open-with-disclosure is + strictly better than today everywhere and overclaims nowhere. +3. **A discarded pass reports the fact every time; the setup advice appears once per + workspace.** Each discarded pass is new information, so suppressing the second one + would hide a real loss. The advice is static configuration guidance, which is what the + existing "say it once" precedent protects against. The short form still names the + variable, because the marker outlives the session that saw the long form. +4. **Cancellation is not a false-✓ vector in any observed form.** See §8. + +## 3. Classification + +One function, one code path. `jq` presence does not change the outcome. It runs on +`PostToolUse` for the two gate tools (default names or the mapped ones) and returns exactly +one class. + +**Input narrowing.** Strip everything up to `"tool_response"` before matching, so nothing +in `tool_input` can be read as a result. `input_field` already uses this defence, stripping +to `"tool_input":{` so a same-named top-level key cannot shadow the real one. The narrowing +matters more here than there: gate prompts routinely quote payload text, so an instruction +containing a marker literal is a realistic input, not a contrived one. + +**The markers are read from the escaped bytes.** The Codex result arrives as a JSON string +*inside* the payload, so the literal characters `\"success\": true` / `\"success\": false` +appear directly in the stream. Nothing decodes the nested string. This is deliberate: a +separate open defect reports the `jq`-free parser breaking on an escaped quote, and +decoding here would walk into it for no gain — the value being read is a boolean. + +**Positional, not "contains".** Classification reads the **first** occurrence of either +polarity after the envelope opening, not any occurrence anywhere. That is what lets a +successful review whose findings quote both literals still classify by its own envelope. + +**Whitespace tolerance.** The matcher tolerates whitespace between key, colon and value. +Only the current serializer's two-space form has been observed; a formatting change should +not silently reclassify every result as unrecognized. + +**Two anchors, provably disjoint.** The envelope anchor requires `"text":"{` at the start +of the content block; the backgrounding notice begins with prose. The two therefore cannot +both match. This is stated here rather than left to the implementation to arrange. + +| Class | Recognized by | Counter | Fingerprint | +|---|---|---|---| +| `success` | first `success` marker after the envelope opening is `true` | bump | store | +| `failure` | first such marker is `false` | no | no | +| `backgrounded` | the notice anchor (§4) | no | no | +| `unrecognized` | none of the above | bump | store | + +**A payload with no `tool_response` at all is `unrecognized`**, not an error. It cannot +occur in production — the field is present in all five captures — but it is what the test +suite sent before this change, and leaving the case unstated would make the classifier's +behaviour there an implementation accident rather than a decision. + +**What the envelope anchor rests on.** `success` being the envelope's first key is observed +`mcp-codex-dev@1.0.1` behaviour — confirmed across all four envelope captures, and matching +the server's own return literal (`dist/tools/codex-exec.js` returns an object whose first +property is `success`, serialized with `JSON.stringify(result, null, 2)`, and insertion +order is preserved). **It is not a JSON guarantee.** Nothing stops a future version +reordering the keys. The degradation is benign by construction: a reordering stops matching +the anchor, so results become `unrecognized`, which is today's behaviour plus a visible +note — never a silent wrong verdict. + +## 4. The backgrounding anchor, and its residual + +The `backgrounded` class is recognized by the harness's own notice text, pinned to its +smallest stable fragment: `still running after`. + +**The benign-degradation argument does NOT extend to this anchor.** Harness prose is not an +API. If the notice is reworded, the payload stops matching, falls to `unrecognized`, and — +on a machine where the environment variable is not set — is counted again. The story's +false ✓ returns there. + +That residual is accepted rather than engineered away, for a stated reason: **the +environment variable is the primary defence, not the anchor.** With +`CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` set, the call never leaves the foreground, so the case +cannot arise at all. The anchor exists to make the unset-variable case visible and +uncounted, and a reworded notice degrades that safety net to at most one disclosure note. + +Recorded plainly so nobody reads the anchor as a guarantee: **a reworded notice on a +machine without the variable set reintroduces an uncounted-pass-counted-as-a-pass.** + +## 5. State effects + +Today the review branch creates the state directory, computes the fingerprint, updates the +fresh-streak counter, writes the fingerprint and bumps the pass count; the exec branch +bumps the Gate-A count. + +- `success` and `unrecognized` — unchanged from today, in full. +- `failure` and `backgrounded` — **skip all of it**, including the directory creation. A + discarded call is a non-event: prior state stays exactly as the last real pass left it. + It is not reset, because resetting would destroy a legitimately earned pass. + +**Classification runs even when the gate is opted out.** `.context/codex-gate.off` +suppresses messages, never state tracking — existing design, so that re-enabling lands on +accurate counters. Skipping classification while off would let a workspace accumulate +uninspected passes and then re-enable into the very false ✓ this change closes. + +## 6. Messages + +Wording is illustrative; the constraint is that each carries cause, effect and — where one +exists — the fix (`docs/prompt-standards.md`). + +- **`failure`, every occurrence.** The call reported failure, so it was not counted and no + fingerprint was stored; an incomplete pass does not count toward the floor. Every time, + because each is a distinct uncounted pass. The agent can already see the error; what it + cannot see is that the counter did not move. +- **`backgrounded`, first time per workspace.** The pass was discarded, not counted: the + call was moved to the background **at the auto-background threshold (120 s by default)**, + so its result never reached the hook. This is a setup gap, not a failed review. Names + `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`, that it needs Claude Code ≥ 2.1.212, and that `0` + disables auto-backgrounding rather than forcing it. +- **`backgrounded`, subsequently.** Short form: discarded, not counted, result never + reached the hook — and the variable name as a pointer. +- **`unrecognized`, once per workspace.** This tool returns a result the hook cannot + verify, so its passes are counted without inspection and may include calls that failed or + reviewed nothing. **No fix instruction**: for a third-party server there is none we can + name, and inventing one would overclaim. + +The threshold is described as a default, never as a constant — it is configurable, so +stating it as fixed would be a claim the code does not make. + +**Marker mechanics** follow the unknown-tool precedent: two new marker files, each written +**only if `emit` succeeded**, so a note suppressed by the off-switch does not burn its +one-shot. The suite already asserts this for the existing note; both new markers get the +same coverage. + +## 7. Testing + +**Sequencing.** The test helper `rev()` currently sends a payload with no `tool_response` +at all, so every existing assertion drives the hook with a result-less payload. It is +updated to carry a real success envelope **before** the classifier lands, so the suite is +green on both sides of the edit and those tests exercise the `success` path rather than +passing by fallback. A test that deliberately wants a result-less payload must say so +explicitly. + +**Coverage**, driven from the captured fixtures rather than hand-written JSON: + +- `success` counts and stores. +- `failure` — both `CODEX_EXECUTION_FAILED` and `CODEX_TIMEOUT` — neither counts nor stores. +- `backgrounded` neither counts nor stores, emits the fact, emits the advice once, and does + not burn its marker when suppressed. +- `unrecognized` counts, stores, and discloses once. +- **Both collision fixtures:** a successful review whose summary quotes both marker + literals must classify `success`; a failed review whose summary quotes `\"success\": + true` must classify `failure`. The second matters more — that is the direction where a + mistake produces the false ✓ this story exists to close. +- `jq`-absent parity for every class, following the suite's existing `jq`-absent pattern. +- The hook exits 0 on every path, including an unreadable result. + +**The `+check` evidence:** the shape-1 and shape-2 tests must be verified failing against +the pre-change hook before the change lands. That is the counterfactual the validation mode +requires, and it is an observation to record, not an assertion to make. + +## 8. Cancellation (story open question 1) + +Four attempts failed to produce a mid-flight abort, and what they did establish: + +- **Esc** rejected the tool call **before dispatch**, twice. No `PostToolUse` fired, and + Codex never started — verified by the absence of any new session under `~/.codex/sessions` + in the window and no tracking directory in the probe repo. +- **`TaskStop` after backgrounding** fired no event, twice. The counter had already moved + at the backgrounding point. + +So **no cancellation path that could be produced fires a hook event**, and cancellation is +not a false-✓ vector in any observed form. The residual, stated precisely: an Esc landing +*after* dispatch was never produced in four attempts, which is weak evidence the harness may +not expose that window. No recognizer is written for it — a matcher built against a shape +nobody has seen fails silently and in the dangerous direction. Should it exist, it lands in +`unrecognized` and counts as today, so there is no regression either way. + +## 9. Documentation and packaging + +- `README.md` `## Setup` documents the variable: its name, Claude Code ≥ 2.1.212, that `0` + disables auto-backgrounding rather than forcing it, and what goes wrong without it. +- `CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` is **not** documented. It exists in the 2.1.220 + binary's environment string table but its behaviour was never exercised; naming the wrong + one of the two in setup text is this repo's docs-drift class. +- The `/workflow-init` preflight is **out of scope** for this story; a follow-up entry is + filed in `todos.md`. +- Plugin manifest version bump (invariant 12) and a `CHANGELOG.md` entry. + +## 10. What this does not do + +- It does not make a counted pass mean Codex read the reviewed bytes. Gate-B validity stays + content-derived; this change only stops calls that returned no review from counting. +- It does not verify third-party envelopes. Those count uninspected, with disclosure. +- It does not survive a reworded backgrounding notice on a machine without the variable set + (§4). +- It does not classify a post-dispatch cancellation, which was never produced (§8). diff --git a/todos.md b/todos.md index e522d9b..7be7c90 100644 --- a/todos.md +++ b/todos.md @@ -166,6 +166,14 @@ backlog. IDs exist to label what profiles produce, so the numbering scheme should meet a real profiled story before it gets a template slot. +- [ ] **`/workflow-init` preflight checks `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`.** The + variable keeps a >120 s gate call in the foreground so its result reaches the hook; + without it a long call is counted at the auto-background threshold having reviewed + nothing. The result-classification story documents it in `README.md` § Setup only, + deliberately — a preflight check is a second surface and was kept out of that diff. + *Trigger: after that story lands* (spec: + `docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md`). + - [ ] **Upstream-report candidate: `claude plugin update `.** The CLI docs document the bare plugin name as a valid argument for `install`/`uninstall`/`update` alike, but `claude plugin update dev-workflow` errors "Plugin 'dev-workflow' not From 696768839c028d1510a67c0fd14638c68577512b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sat, 1 Aug 2026 11:02:04 +0200 Subject: [PATCH 04/10] docs(spec): close Gate A on the result-classification design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight Gate-A passes (17, 13, 14, 4, 7, 7, 6, 6 MAJOR), every one validated. Closed on judgement, NOT on a NO FINDINGS pass -- recorded as such in .context/codex-reviews/gate-a-spec-CLOSURE.md, which also carries the obligations the plan must check off. Design changes across the cycle: a five-class classifier (success, failure, backgrounded, no-result, unrecognized) with the class table as the single normative definition; classification collapsed from two representations to one, with the jq path verifying its re-encoded block occurs exactly once inside the located span so both parser environments read the same bytes or refuse together; no-result stated by its complement so unanticipated shapes fail closed; malformed outer JSON reclassified as unrouteable rather than a class, since the hook derives its own routing from that document. Story amended four times, each recorded inline with what it replaced: §2 at passes 2 and 5, §3 criteria at passes 4 and 8. §2 is now marked a summary deferring to the spec, so a future amendment has one target. todos.md: two trigger-gated rows for pre-existing conditions this design states as contracts rather than fixes -- unserialized counter mutation, and the hook's trust of a repository-controlled .context/. Gate B N/A: no code or prompt paths staged. Invariant checks re-run clean. --- ...iled-codex-call-counts-as-a-pass-design.md | 714 ++++++++++++++---- ...ailed-codex-call-counts-as-a-pass-story.md | 48 +- todos.md | 20 + 3 files changed, 608 insertions(+), 174 deletions(-) diff --git a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md index 7f4e626..7c8559f 100644 --- a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md +++ b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md @@ -3,10 +3,12 @@ **Date:** 2026-07-31 **Story:** `docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md` — read its header for the current risk/security/validation profile; this spec deliberately -does not copy those values, so there is one writable copy. +does not copy those values, so there is one writable copy. (The story was amended twice during this +design's Gate A — §2 at pass 2 and again at pass 5, and the §3 criteria at pass 4. Each +amendment is recorded inline in the story with what it replaced.) -**Evidence:** `.context/probe-payloads/` (untracked; `INDEX.md` records provenance and the -counter effect per shape). Five captured `PostToolUse` payloads, all real. +**Evidence:** captured `PostToolUse` payloads, currently in `.context/probe-payloads/`. +§7 moves them to a tracked, shipped home. ## 1. What this changes @@ -15,204 +17,586 @@ Codex calls that reviewed nothing, because nothing in the `PostToolUse` path ins result before counting. This adds one classifier that reads the result and decides whether the call earned a pass. -The problem statement, the measured counter movements and the four observed payload shapes -are in the story. This spec covers only the design. - ## 2. Settled decisions -Four questions were answered by the human before design, and their reasons are recorded -here because a later reader will otherwise re-litigate them: - -1. **Classification keys on the envelope, not the tool name.** A tool mapped via - `.context/codex-gate.tools` is in scope. -2. **An unrecognized envelope fails OPEN** — it counts, as today, plus a once-per-workspace - disclosure. Fail-closed would zero the counters permanently for every third-party - server with no fix available to the user, which is the hook's own worst failure mode - ("reviews run, counters stay 0, and the STOP fires on every commit forever — which - trains the user to ignore the hook", `codex-gate.sh`). Fail-open-with-disclosure is - strictly better than today everywhere and overclaims nowhere. -3. **A discarded pass reports the fact every time; the setup advice appears once per - workspace.** Each discarded pass is new information, so suppressing the second one - would hide a real loss. The advice is static configuration guidance, which is what the - existing "say it once" precedent protects against. The short form still names the - variable, because the marker outlives the session that saw the long form. -4. **Cancellation is not a false-✓ vector in any observed form.** See §8. +Decided by the human; recorded with their reasons so a later reader does not re-litigate +them. + +1. **Classification keys on the result envelope, not the tool name.** A tool mapped via + `.context/codex-gate.tools` is in scope — **within the reachable namespace**. The hook is + invoked by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming + a tool outside `mcp__codex__*` can never fire, and always could not. The matcher is left + as it is: broadening it would start a hook process on every MCP tool call in every + adopted workspace, which is a real cost for a case with a one-line remedy — register the + third-party server under the server name `codex`, which places its tools in the reachable + namespace. The boundary and that remedy are documented where the mapping is documented, + and the mapped-tool tests use `codex`-namespaced names, which is the path that can + actually fire. +2. **An unrecognized envelope FAILS OPEN** — it counts, plus a once-per-workspace + disclosure. Fail-closed would zero the counters permanently for every third-party server + with no fix available to the user, which is the hook's own worst failure mode ("reviews + run, counters stay 0, and the STOP fires on every commit forever — which trains the user + to ignore the hook", `codex-gate.sh`). +3. **An unreadable result FAILS CLOSED**, as a class distinct from `unrecognized`. + Decision 2's rationale is third-party envelope *variance inside a result*: a result the + hook can see but cannot interpret. A payload from which no result text can be obtained is + a different thing — there is nothing to interpret — so fail-open's rationale does not + reach it. Note the narrower premise this rests on: for the *pinned* server the shape is + unproducible, but a mapped third-party tool may legitimately return empty or non-text + content (§3.1). Fail-closed holds either way, because an unreadable result is not + evidence of a review; only the diagnosis differs (§6). +4. **A discarded pass reports the fact on every occurrence; the setup advice appears once + per workspace**, and the short form still names the variable, because the marker outlives + the session that saw the long form. +5. **Cancellation is not a false-✓ vector in any observed form**; no recognizer is written + for an unobserved shape (§8). ## 3. Classification -One function, one code path. `jq` presence does not change the outcome. It runs on -`PostToolUse` for the two gate tools (default names or the mapped ones) and returns exactly -one class. - -**Input narrowing.** Strip everything up to `"tool_response"` before matching, so nothing -in `tool_input` can be read as a result. `input_field` already uses this defence, stripping -to `"tool_input":{` so a same-named top-level key cannot shadow the real one. The narrowing -matters more here than there: gate prompts routinely quote payload text, so an instruction -containing a marker literal is a realistic input, not a contrived one. - -**The markers are read from the escaped bytes.** The Codex result arrives as a JSON string -*inside* the payload, so the literal characters `\"success\": true` / `\"success\": false` -appear directly in the stream. Nothing decodes the nested string. This is deliberate: a -separate open defect reports the `jq`-free parser breaking on an escaped quote, and -decoding here would walk into it for no gain — the value being read is a boolean. - -**Positional, not "contains".** Classification reads the **first** occurrence of either -polarity after the envelope opening, not any occurrence anywhere. That is what lets a -successful review whose findings quote both literals still classify by its own envelope. - -**Whitespace tolerance.** The matcher tolerates whitespace between key, colon and value. -Only the current serializer's two-space form has been observed; a formatting change should -not silently reclassify every result as unrecognized. - -**Two anchors, provably disjoint.** The envelope anchor requires `"text":"{` at the start -of the content block; the backgrounding notice begins with prose. The two therefore cannot -both match. This is stated here rather than left to the implementation to arrange. +Runs on `PostToolUse` for the two gate tools (default names or mapped ones) and returns +exactly one of five classes. + +### 3.1 Obtaining the result text + +**Which block, settled here.** The classifier reads the **first array element of +`tool_response` whose `type` is `text`**, and its `text` value. Not element `[0]`: a mapped +tool may legitimately return an image or other block first, and indexing blindly would miss +the result or feed a non-text block to the matcher. + +**Locating is where the two environments differ; matching is not.** Both paths produce the +**same representation** — the block's text in its **escaped** JSON encoding — and hand it to +one matcher (§3.2). There is no decoded-vs-escaped equivalence to maintain, no parity rule +over matching, and no second implementation of the classification contract. + +- **With `jq`:** locate structurally, then **re-encode to the escaped form**. The selector + must implement the structural contract rather than assume it: gate on `tool_response` + being an **array**, skip elements that are not objects, take the first whose `type` is + exactly `text`, and require its `text` to be a **string** — a bare + `.tool_response[]? | select(.type=="text") | .text` does none of these (`[]?` iterates an + object's *values*, `select` errors on a non-object element, and `.text` goes untyped). +- **Without `jq`:** an escape-aware scan locates the same block; what it yields is already + the escaped form. + +**Byte-position heuristics are unsafe in both directions**, so neither is used: a greedy +strip anchors on the *last* `tool_response` match, which a result quoting the key can +hijack; a first-match strip anchors on the *earliest*, which `tool_input` can hijack, and +this repo's own gate prompts quote payload text. + +**The matcher always consumes the payload's own bytes.** After locating, the `jq` path +re-encodes the block and finds that byte sequence in the payload; **the matcher then reads the +payload at that position**, never the string `jq` produced. Two conditions must hold or the +payload is **`unrecognized`**: the re-encoded block must occur **exactly once** in the +payload, and the occurrence must lie **within the located `tool_response` span**. + +Both conditions are load-bearing. "Appears somewhere" is not enough: a canonical copy sitting +in `tool_input` would satisfy it for a response that is *not* canonical, and a non-canonical +escape appearing after an otherwise-canonical marker would leave the two environments +disagreeing about the same bytes. Requiring a unique in-span occurrence, and feeding the +matcher the original bytes, makes both environments read the same input or refuse together — +so the divergence is *removed* rather than documented. + +**Mislocation is safe by construction.** If the block cannot be resolved unambiguously — the +key appears more than once and depth cannot be established, or the canonical-form check above +fails — the payload is **`unrecognized`**, which counts and discloses. Locator uncertainty +never produces a *wrong verdict*; it produces a counted pass whose disclosure is **attempted +and normally persisted**. Where both the emit and the pending write fail (§5.2) that count is +silent, so "never silent" would be false: the verdict is never wrong, the disclosure is +best-effort. + +**The class definitions are normative in the §3.3 table and stated nowhere else**, so an +edit cannot desynchronize them. What §3.1 contributes is the input those definitions consume: +the located block, or a determination that none can be had. The one boundary worth naming +here, because it is a property of *locating* rather than of any class: an unambiguous +determination that there is nothing usable is a different outcome from not being able to +determine anything, and §3.3 routes them to different classes — fail-closed and fail-open +respectively. + +**Why fail-closed is right here, at the precision the evidence supports.** For the pinned +server these shapes are unproducible, so each indicates a payload-contract change. That +premise does **not** generalize: a mapped third-party tool may legitimately return empty +content, an image block, or another non-text shape. Fail-closed holds either way — an +unreadable result is not evidence of a review — but the *diagnosis* must not assume the +harness is at fault (§6). + +### 3.2 Reading the marker + +**One matcher, one representation.** On the escaped text produced by §3.1, the marker reads +`\"success\": true` or `\"success\": false`. + +**Immediately-first, not merely first.** It must be the envelope's first property, +immediately after the encoded object opening, allowing encoded JSON whitespace. "First +occurrence anywhere after the opening" would still match under key reordering, contradicting +the benign-degradation claim below; requiring immediate-first position is what makes +reordering degrade to `unrecognized` rather than to a wrong verdict. + +**Whitespace tolerance** between key, colon and value. Only the current serializer's +two-space form has been observed; a formatting change should not silently reclassify. + +**Neither path parses the result text as nested JSON.** `jq` decodes it as a string value +and re-encodes it; the scan never decodes at all. What neither does is run a JSON parser +*into* the result string, which is what the open escaped-quote defect makes unsafe. + +### 3.3 The five classes + +**This table is the single normative definition of the five classes.** Every other section +refers to it rather than restating it. | Class | Recognized by | Counter | Fingerprint | |---|---|---|---| -| `success` | first `success` marker after the envelope opening is `true` | bump | store | -| `failure` | first such marker is `false` | no | no | -| `backgrounded` | the notice anchor (§4) | no | no | -| `unrecognized` | none of the above | bump | store | - -**A payload with no `tool_response` at all is `unrecognized`**, not an error. It cannot -occur in production — the field is present in all five captures — but it is what the test -suite sent before this change, and leaving the case unstated would make the classifier's -behaviour there an implementation accident rather than a decision. +| `success` | located block's immediate-first property is `success: true` | bump | store | +| `failure` | located block's immediate-first property is `success: false` | no | no | +| `backgrounded` | the notice anchor (§4), at start of the located block | no | no | +| `no-result` | an **unambiguous** determination that no located block yields a non-blank string: `tool_response` absent, `null`, empty array, non-array container, non-object elements, no `text`-type element, `text` not a string, or blank text | no | no | +| `unrecognized` | everything else — a located block matching no anchor, **and every case where locating itself is uncertain**: ambiguous boundary, repeated depth-1 `tool_response` key, or a failed uniqueness/in-span check (§3.1) | bump | store | + +Stating `no-result` by its complement is deliberate: a hooks-API shape nobody anticipated +lands in the **fail-closed** class, which is the direction that matters, since `unrecognized` +counts. + +**"Blank" is defined on the shared representation, not semantically**, or the two environments +could disagree on a state-changing boundary. The located block is blank when its **escaped +bytes** contain nothing but ASCII space and the two-byte escapes `\n`, `\t`, `\r`. A +Unicode-escaped space (`\u0020`) is *not* blank by this rule, and does not need to be: such a +block fails the canonical-form check first. **Order is fixed** — canonical-form validation +(§3.1) runs *before* the blank test, so every encoding `jq` would normalize has already been +routed to `unrecognized` and never reaches this comparison. §11 item 3 carries the full +encoding table. + +**Malformed outer JSON is not a class at all, and pass 6 got this wrong.** The hook derives +`hook_event_name` and `tool_name` from the same document; if it is malformed, the hook cannot +establish that this was a gate call, so it can neither classify it nor address a message to +it. Assigning it `no-result` would demand an every-occurrence gate diagnosis on an invocation +that might have been a `Bash` PreToolUse. A payload the hook cannot **route** therefore +touches no state and emits nothing, exiting 0 — the pre-existing behaviour for anything it +cannot parse, and unchanged here. + +**One outcome is settled here**, because it is a classification rather than scanner mechanics: +a **repeated depth-1 `tool_response` key** is **`unrecognized`** (the locator is genuinely +ambiguous). The plan defines only how the scanner *recognizes* that state, not what it means. + +**Precedence, total and explicit:** `no-result` → `backgrounded` → envelope polarity → +`unrecognized`. Every payload reaches exactly one class, because `unrecognized` is the +terminal default and `no-result` is decided structurally before any text matching. **What the envelope anchor rests on.** `success` being the envelope's first key is observed -`mcp-codex-dev@1.0.1` behaviour — confirmed across all four envelope captures, and matching -the server's own return literal (`dist/tools/codex-exec.js` returns an object whose first -property is `success`, serialized with `JSON.stringify(result, null, 2)`, and insertion -order is preserved). **It is not a JSON guarantee.** Nothing stops a future version -reordering the keys. The degradation is benign by construction: a reordering stops matching -the anchor, so results become `unrecognized`, which is today's behaviour plus a visible -note — never a silent wrong verdict. +`mcp-codex-dev@1.0.1` behaviour, verified two ways: every captured envelope shows it, and +every tool-level return in the pinned server leads with `success` — `codex-exec.js` (success +and catch returns) and `codex-review.js` at three sites (catch, `full`-mode, single +reviewer) — each serialized with `JSON.stringify(result, null, 2)`, which preserves +insertion order. **It is not a JSON guarantee.** A future reordering stops matching, so +results become `unrecognized`: today's behaviour plus a visible note, never a silent wrong +verdict. ## 4. The backgrounding anchor, and its residual -The `backgrounded` class is recognized by the harness's own notice text, pinned to its -smallest stable fragment: `still running after`. +**The exact pattern**, anchored at the **start of the result text**: the text begins with +`MCP tool "`, and the segment `" is still running after ` follows before any newline. The +variable parts are therefore explicit — the quoted tool name (`codex/exec`, `codex/review`, +or any mapped name), the threshold value, and the task id are all outside the matched +anchor, so the anchor covers both gate tools and any configured threshold. A bare fragment +such as `still running after` is *not* the anchor: it can legitimately occur inside a +result's own summary — a review discussing this mechanism would contain it — and would let +one payload satisfy two anchors. + +**Precedence, stated defensively:** `backgrounded` is tested before the envelope polarity, +and only at start-of-text. A genuine notice never begins with an encoded envelope opening +and a genuine envelope never begins with the notice prose, so the two are disjoint by +construction — and the ordering keeps the outcome defined even if a future payload violates +that. **The benign-degradation argument does NOT extend to this anchor.** Harness prose is not an -API. If the notice is reworded, the payload stops matching, falls to `unrecognized`, and — -on a machine where the environment variable is not set — is counted again. The story's -false ✓ returns there. +API. If the notice is reworded, the payload stops matching, falls to `unrecognized`, and — on +any runtime where auto-backgrounding is still **effective** — is counted again. That is +broader than "the variable is unset": it also covers a Claude Code older than 2.1.212, which +does not read the variable at all, and a *positive* value shorter than the call, which +backgrounds anyway. The story's false ✓ returns there. -That residual is accepted rather than engineered away, for a stated reason: **the -environment variable is the primary defence, not the anchor.** With -`CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` set, the call never leaves the foreground, so the case -cannot arise at all. The anchor exists to make the unset-variable case visible and -uncounted, and a reworded notice degrades that safety net to at most one disclosure note. +That residual is accepted because **the environment variable is the primary defence, not the +anchor**: configured correctly the call never leaves the foreground, so the case cannot +arise. The anchor is a safety net for the misconfigured case. -Recorded plainly so nobody reads the anchor as a guarantee: **a reworded notice on a -machine without the variable set reintroduces an uncounted-pass-counted-as-a-pass.** +Recorded plainly: **a reworded notice, on any runtime where auto-backgrounding still takes +effect, reintroduces an uncounted pass being counted.** ## 5. State effects -Today the review branch creates the state directory, computes the fingerprint, updates the -fresh-streak counter, writes the fingerprint and bumps the pass count; the exec branch -bumps the Gate-A count. - -- `success` and `unrecognized` — unchanged from today, in full. -- `failure` and `backgrounded` — **skip all of it**, including the directory creation. A - discarded call is a non-event: prior state stays exactly as the last real pass left it. - It is not reset, because resetting would destroy a legitimately earned pass. - -**Classification runs even when the gate is opted out.** `.context/codex-gate.off` -suppresses messages, never state tracking — existing design, so that re-enabling lands on -accurate counters. Skipping classification while off would let a workspace accumulate -uninspected passes and then re-enable into the very false ✓ this change closes. +### 5.1 Gate-pass state + +State effects are a **column of the §3.3 table**, which is normative. This section explains +what those column values do; it does not restate which class gets which. + +- Classes the table marks bump/store — today's behaviour, unchanged and in full. +- Classes the table marks no/no — **no gate-pass state is written or modified.** + Every pre-existing pass-state file keeps its exact prior contents: `passCount`, + `freshCount`, `passCountA`, and the fingerprint. A discarded call is a non-event; it is + not reset, because resetting would destroy a legitimately earned pass. + +**Concurrency contract, stated rather than changed.** Counter mutation stays +read-modify-write and unserialized, and the fingerprint, fresh-count and pass-count writes +stay independent. Concurrent `PostToolUse` events can therefore lose an increment or expose +a mixed snapshot. This **predates this change** and applies to every counter; the change +adds files with the same property, not new exposure, so fixing it here would be an +inconsistent partial repair. Filed in `todos.md` with a trigger. + +**Trust contract, likewise stated.** The hook creates and truncates files inside a +repository-controlled `.context/`, following symlinks, and does not defend against a hostile +workspace — true today of every state file it writes. Filed in `todos.md` with a trigger. + +### 5.2 Diagnostic state + +Distinct from gate-pass state, and the only state a discarded call may write. Three paths: + +| Path | Meaning | +|---|---| +| `.context/codex-gate.bgAdvice` | the long backgrounding advice has been shown | +| `.context/codex-gate.unverified` | the unrecognized-envelope disclosure has been shown | +| `.context/codex-gate.unverifiedPending` | a disclosure is owed but was suppressed | + +A discarded call may create `.context/` and a marker on a best-effort basis — otherwise the +first-ever discarded call in a fresh workspace could never record that its advice was shown, +and the long advice would repeat forever. It must never create, clear or rewrite a +**pass-state** file. Marker writes stay best-effort so a failure cannot break invariant 1. + +**Lifecycle.** Markers are workspace-scoped and are *not* cleared by a commit reset, which +closes a Gate-B cycle rather than changing what the tool surface is capable of. `reset_all()` +in the suite clears them, or one-shot tests become order-dependent. + +**State transitions for the disclosure:** + +| Current | Event | Result | +|---|---|---| +| absent | `unrecognized`, gate on, emit succeeds | `unverified` written | +| absent | `unrecognized`, gate on, **emit fails** | `unverifiedPending` written (best-effort) | +| absent | `unrecognized`, gate off (emit suppressed) | `unverifiedPending` written | +| pending | any unsuppressed hook event | disclosure emitted, then `unverified` written and pending cleared | +| pending | emit fails, or `unverified` write fails | pending **retained** | +| shown | `unrecognized` again | nothing emitted, nothing written | + +**Pending is cleared only after the shown-marker write succeeds.** A duplicated disclosure +is strictly better than a lost one, so the failure direction repeats rather than drops. If +both writes fail persistently the disclosure repeats every time — noisy, and the safe +direction. + +**One residual is NOT covered by that, and is accepted explicitly — in both directions.** +It applies whenever the disclosure is not delivered *and* the pending write also fails: +either the gate was off and suppressed the emit, or the gate was on and the emit failed. The +gate-on path added above narrows the window but does not close it, so the claim is that the +emit-failure fix gives the disclosure a durable home when the write succeeds — not that it +removes the residual. Concretely: if an `unrecognized` call arrives, its disclosure is not +delivered for either reason, and the *pending* write itself fails, the pass has +already been counted and nothing durable records that a disclosure is owed. Re-enabling then +yields exactly the undisclosed satisfied count that pending exists to prevent. Two policies +were available: withhold the count until pending persistence succeeds, or accept the loss. +**The count is not withheld**, because doing so would make an unwritable `.context/` silently +stop counting legitimate passes — the permanent-zero-counter failure decision 2 exists to +avoid, arriving by a different route. So this is an accepted residual, not a covered case: +**an unrecognized call while off, whose pending write fails, is counted and never +disclosed.** It requires a write failure in a directory the hook otherwise depends on, which +is why it is accepted rather than engineered around. + +**Pending is not only about the off-switch.** A failed emit while the gate is *on* leaves the +same debt: the call is counted and the disclosure was never delivered. Without a pending +write there, an unrelated later event has nothing to flush, and the workspace can reach a +satisfied count made of uninspected calls with the once-per-workspace disclosure never shown +— which is decision 2's guarantee broken through a path the gate-off reasoning never +covered. So a failed emit takes the same best-effort pending write as suppression does. + +**Why pending exists.** While the gate is off, an `unrecognized` call still counts, but its +disclosure cannot be shown. Writing the shown-marker anyway would consume the one-shot for a +message nobody saw; writing nothing would let a workspace re-enable into a satisfied count +composed entirely of uninspected calls with no disclosure ever shown. + +**Classification runs even when the gate is opted out** — `.context/codex-gate.off` +suppresses messages, never state tracking, so re-enabling lands on counters carrying the same +semantics as if the gate had been on throughout. Not "accurate" in the sense of counting only +earned reviews: an `unrecognized` call counts while off exactly as it does while on, so a +re-enabled workspace can hold an uninspected counted pass. Pending state exists to disclose +those (subject to §5.2's residual); the counters themselves are never evidence that a review +happened. ## 6. Messages -Wording is illustrative; the constraint is that each carries cause, effect and — where one -exists — the fix (`docs/prompt-standards.md`). - -- **`failure`, every occurrence.** The call reported failure, so it was not counted and no - fingerprint was stored; an incomplete pass does not count toward the floor. Every time, - because each is a distinct uncounted pass. The agent can already see the error; what it - cannot see is that the counter did not move. -- **`backgrounded`, first time per workspace.** The pass was discarded, not counted: the - call was moved to the background **at the auto-background threshold (120 s by default)**, - so its result never reached the hook. This is a setup gap, not a failed review. Names - `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`, that it needs Claude Code ≥ 2.1.212, and that `0` - disables auto-backgrounding rather than forcing it. -- **`backgrounded`, subsequently.** Short form: discarded, not counted, result never - reached the hook — and the variable name as a pointer. -- **`unrecognized`, once per workspace.** This tool returns a result the hook cannot - verify, so its passes are counted without inspection and may include calls that failed or - reviewed nothing. **No fix instruction**: for a third-party server there is none we can - name, and inventing one would overclaim. - -The threshold is described as a default, never as a constant — it is configurable, so -stating it as fixed would be a claim the code does not make. - -**Marker mechanics** follow the unknown-tool precedent: two new marker files, each written -**only if `emit` succeeded**, so a note suppressed by the off-switch does not burn its -one-shot. The suite already asserts this for the existing note; both new markers get the -same coverage. +**Delivery is best-effort, and the categorical wording below is scoped to that.** "Once per +workspace", "every occurrence" and "nothing is dropped" describe the sequential, no-failure +case. They are not guarantees: a suppressed gate, a failed stdout, a failed marker write, the +accepted residual in §5.2, and concurrent check-emit-write sequences each permit a message to +be lost or repeated. The accepted directions are stated where each arises — duplication is +preferred to loss throughout. What *is* unconditional is narrower than "an unearned pass is +never counted", and stating it correctly matters because this paragraph exists to calibrate +expectations: a call classified `failure`, `backgrounded` or `no-result` is **never** counted, +message or no message. A genuinely unearned call that classifies `unrecognized` **is** counted +— decision 2 working as intended, and the accepted false-pass residual rather than an +exception to it. + +**One emit per hook invocation.** `emit` writes a single hook JSON document, so two messages +cannot be two emits. When an invocation owes both a pending disclosure and a per-occurrence +message, they are **composed into one emit** — disclosure first, then the per-occurrence +message — with both `additionalContext` bodies joined and both `systemMessage` bodies +joined. Nothing is dropped and nothing is deferred; a deferred message would collide again +on the next event. + +**Field split.** `additionalContext` (model-facing) carries the consequence for the gate. +`systemMessage` (user-visible) carries any operator action, because the operator is who can +perform it. + +- **`failure`, every occurrence.** Not counted, no fingerprint stored; an incomplete pass + does not count toward the floor. Every time, because each is a distinct uncounted pass — + the agent can see the error, but not that the counter did not move. +- **`backgrounded`, first time per workspace.** Discarded, not counted: the call was moved + to the background **at the auto-background threshold (120 s by default)**, so its result + never reached the hook. A setup gap, not a failed review. The `systemMessage` names the + operator action: set `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` **in the environment Claude Code + is launched from, then restart it** — the variable is read at process start, so exporting + it inside a tool shell cannot affect the running parent. States that `0` disables + auto-backgrounding, and that a *positive* value must exceed the longest expected gate call, + since a positive value shorter than the call still backgrounds. Requires Claude Code + ≥ 2.1.212. +- **`backgrounded`, subsequently.** Short form: discarded, not counted, result never reached + the hook — plus the variable name as a pointer. +- **`no-result`, every occurrence.** No tool result was obtainable, so the pass was not + counted. It names **two** causes rather than assuming the harness is at fault: a hooks-API + payload contract change (check the Claude Code version and report it — for the pinned + server this shape is unreachable), **or** a mapped third-party tool returning empty or + non-text content, which is legitimate for that tool and simply unreadable as a gate result. + **The tool name alone cannot tell them apart**, and the message must not pretend + otherwise: decision 1's own remedy is registering a third-party server *as* `codex`, which + makes the names identical. The checks it gives instead are `.context/codex-gate.tools` for + an active mapping, and the effective MCP registration and pinned server version in + `.mcp.json`. Where provenance stays unknown the message says so and offers both remedies + rather than asserting one. +- **`unrecognized`, once per workspace.** Enumerates its causes, each with what can be done + about it: + - a pinned-server envelope whose key order or formatting changed — check the server + version against `.mcp.json`; fixable by pinning back; + - **a reworded harness backgrounding notice** (§4) — the call was backgrounded and the + anchor no longer matches, so the remedy is the same + `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` guidance the `backgrounded` message carries; + - a mapped third-party tool whose envelope this hook cannot read — **no user-side fix**; + - an ambiguous or non-canonically-encoded payload (§3.1) — this resolves the same way in + both environments, so installing `jq` does **not** change it; + - a hook parser defect — **no operator fix**; the check is to run the hook against the + captured payload and report the mismatch. + + Two causes lack an operator fix, and the message says so rather than claiming a single one + does. **The list is not claimed to be exhaustive** — `unrecognized` is the terminal default + class, so any future unmatched shape lands here too; the message says these passes are + counted without inspection and may include calls that failed or reviewed nothing. + + Including the reworded-notice cause matters because §4 accepts that residual: without it, + the one case where the hook silently counts a backgrounded call would also be the case + where it withholds the setting that prevents recurrence. + +The threshold is always described as a default, never a constant. + +**Prompt conformance.** These are hook messages, so invariant 11 applies in full: every +final string — long, short, pending, and per-occurrence — is reviewed against **all 12** +items of `docs/prompt-standards.md`, not item 10 alone. The suite pins both output fields +with exact golden assertions, following the existing `matches exactly` tests, because a +clause grep cannot catch a negation or semantic drift. + +**Marker mechanics.** A marker is written only if `emit` succeeded, where **success means a +complete hook JSON document was written to stdout** — not merely that the function was +called. Otherwise a note suppressed or truncated would burn its one-shot. + +**This requires changing `emit` itself**, which today returns 0 unconditionally after its +output command: it must propagate the status of the write. Without that change the +definition above is unimplementable and the pass-2 fix it came from would be prose only. A +closed or failing stdout must therefore leave the marker unwritten and any pending state +retained — while the hook still exits 0, since invariant 1 is about the hook's exit status, +not about whether a message reached anyone. ## 7. Testing -**Sequencing.** The test helper `rev()` currently sends a payload with no `tool_response` -at all, so every existing assertion drives the hook with a result-less payload. It is -updated to carry a real success envelope **before** the classifier lands, so the suite is -green on both sides of the edit and those tests exercise the `success` path rather than -passing by fallback. A test that deliberately wants a result-less payload must say so -explicitly. - -**Coverage**, driven from the captured fixtures rather than hand-written JSON: - -- `success` counts and stores. -- `failure` — both `CODEX_EXECUTION_FAILED` and `CODEX_TIMEOUT` — neither counts nor stores. -- `backgrounded` neither counts nor stores, emits the fact, emits the advice once, and does - not burn its marker when suppressed. -- `unrecognized` counts, stores, and discloses once. -- **Both collision fixtures:** a successful review whose summary quotes both marker - literals must classify `success`; a failed review whose summary quotes `\"success\": - true` must classify `failure`. The second matters more — that is the direction where a - mistake produces the false ✓ this story exists to close. -- `jq`-absent parity for every class, following the suite's existing `jq`-absent pattern. -- The hook exits 0 on every path, including an unreadable result. - -**The `+check` evidence:** the shape-1 and shape-2 tests must be verified failing against -the pre-change hook before the change lands. That is the counterfactual the validation mode -requires, and it is an observation to record, not an assertion to make. +### 7.1 Fixtures + +Fixtures move to `plugins/dev-workflow/hooks/fixtures/`, tracked and shipped, resolved +relative to the test script's own directory so the suite works from the repo and from an +installed plugin copy alike. Invariant 12 covers plugin paths, so a fixture change requires a +version bump like any other. + +**They are sanitized captures, and a `README` in that directory says so.** Machine-specific +metadata irrelevant to classification — `transcript_path`, session identifiers, `cwd` — is +neutralized. The `tool_response` block is kept **byte-exact**, because that is the object +under test. The README records each fixture's provenance and the fact of sanitization, so +"captured" stays an honest word. + +The set covers **both gate tools**: every envelope captured so far is from `exec`, while the +classifier also governs `review` and Gate-B fingerprint storage, so real `review` envelopes +are captured for it. **A review response embeds review content** — file paths, code excerpts, +findings — and `tool_response` must stay byte-exact, so sanitization cannot clean it after the +fact. The review fixture is therefore captured **against a disposable synthetic repository +with controlled, non-sensitive content**, and the whole fixture is read end to end before it +ships. Capturing one from real work and scrubbing it afterwards is the shape to avoid: +byte-exactness and redaction are mutually exclusive. The source-level verification in §3.3 is corroboration, not a +substitute. + +### 7.2 Sequencing + +Existing helpers send payloads with no `tool_response` — `rev()`, `codextool()` and several +direct `exec` payloads. Under this design those become `no-result` and stop counting, so +**every existing counting call site is updated to carry a real success envelope before the +classifier lands**, not `rev()` alone. The suite is green on both sides of the edit. A +deliberately result-less helper is added for the `no-result` tests, so that case is stated +rather than inherited. + +### 7.3 Coverage + +- `success` counts and stores — **and asserts no disclosure marker was created**. Without + that, the test cannot distinguish `success` from `unrecognized`, whose counter and + fingerprint effects are identical, and would pass against a malformed envelope classified + by fallback. +- `failure` (`CODEX_EXECUTION_FAILED`, `CODEX_TIMEOUT`), `backgrounded`, and `no-result` + each neither count nor store. +- **`no-result` shape coverage**: absent field, `null`, empty array, non-array container, + non-object elements, array without a `text`-type block, non-string `text`, and empty or + whitespace-only `text`, in both parser environments. +- **Block selection**: a response whose first element is a non-text block followed by a real + text block must classify from the text block, in both parser environments — the case that + distinguishes "first `text`-type element" from "element `[0]`". +- **Writer-failure coverage**: with stdout closed or failing, in both parser modes, the hook + exits 0, writes no shown-marker, retains any existing pending state, and **creates pending + when none existed** (the gate-on failed-emit transition). Marker-write failure alone is not + sufficient coverage — it leaves the `emit`-status path untested. +- **Seeded-state preservation, both gates**: for each discarded class, seed a legitimate + `passCount`, `freshCount`, `passCountA` and fingerprint, then assert all are byte-identical + afterwards — including **Gate-A `passCountA` for discarded `exec` calls**, and for mapped + exec names. Empty-state assertions alone would pass an implementation that clears earned + state. +- `unrecognized` counts, stores, discloses once; the pending path is asserted across an + off→on transition, including retention when the shown-write fails. +- **Composition**: an invocation owing both a pending disclosure and a per-occurrence message + emits exactly one valid hook JSON document containing both. +- **Mapped tools**: captured success and failure envelopes through mapped `exec` and `review` + names, asserting the same classes and state effects as the defaults. Without this an + implementation keying on default names only would satisfy the suite while violating + decision 1. +- **Boundary and collision fixtures**: a `tool_input` containing `"tool_response"` before the + real field; a result quoting `"tool_response"` after it; duplicate outer keys; a successful + review quoting both marker literals (must classify `success`); a failed review quoting + `\"success\": true` (must classify `failure`). The failure-direction cases matter most — + that is where a mistake produces the false ✓. +- **Extraction parity, asserted on the matcher's input.** Every fixture runs through both + locating paths, and the escaped block handed to the matcher must be **byte-identical** + between them. Asserting only the final class would let two locating bugs cancel out and + report a pass; asserting the input catches a locating divergence where it lives. The + single matcher then needs no parity assertions of its own — there is only one of it. +- The former divergence cases (ambiguous boundary, Unicode-escaped marker, + non-canonical encoding) now assert **`unrecognized` in BOTH environments** — the + canonical-form check makes the `jq` path reach the raw scan's verdict. A test asserting a + precise class with `jq` would pin the very divergence §3.1 removes. +- Golden assertions on both output fields for every message. +- The hook exits 0 on every path, including an unreadable result and a marker-write failure. + +### 7.4 Validation evidence + +- **The `+check` counterfactual:** the `failure`-class tests must be verified failing against + the pre-change hook before the change lands — an observation to record, not an assertion to + make. +- **The named verification**, required by the story's profile: a post-change rerun of the + probe methodology, recording the counter and fingerprint reading for each of both failure + envelopes, `backgrounded` with the variable absent, `backgrounded` prevented with the + variable set, and one genuine pass that still counts. The evidence entry lands in the + commit body per CLAUDE.md §5. ## 8. Cancellation (story open question 1) -Four attempts failed to produce a mid-flight abort, and what they did establish: +Four attempts failed to produce a mid-flight abort. Established: -- **Esc** rejected the tool call **before dispatch**, twice. No `PostToolUse` fired, and - Codex never started — verified by the absence of any new session under `~/.codex/sessions` - in the window and no tracking directory in the probe repo. -- **`TaskStop` after backgrounding** fired no event, twice. The counter had already moved - at the backgrounding point. +- **Esc** rejected the call **before dispatch**, twice: no `PostToolUse` fired, and Codex + never started — verified by the absence of any new session under `~/.codex/sessions` in + the window and no tracking directory in the probe repo. +- **`TaskStop` after backgrounding** fired no event, twice; the counter had already moved at + the backgrounding point. -So **no cancellation path that could be produced fires a hook event**, and cancellation is -not a false-✓ vector in any observed form. The residual, stated precisely: an Esc landing -*after* dispatch was never produced in four attempts, which is weak evidence the harness may -not expose that window. No recognizer is written for it — a matcher built against a shape -nobody has seen fails silently and in the dangerous direction. Should it exist, it lands in -`unrecognized` and counts as today, so there is no regression either way. +So **no cancellation path that could be produced fires a hook event.** The residual: an Esc +landing *after* dispatch was never produced in four attempts, which is weak evidence the +harness may not expose that window. No recognizer is written for it — a matcher built +against an unobserved shape fails silently and in the dangerous direction. Should it exist, +it lands in `unrecognized` and counts as today, so there is no regression. ## 9. Documentation and packaging -- `README.md` `## Setup` documents the variable: its name, Claude Code ≥ 2.1.212, that `0` - disables auto-backgrounding rather than forcing it, and what goes wrong without it. -- `CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` is **not** documented. It exists in the 2.1.220 - binary's environment string table but its behaviour was never exercised; naming the wrong - one of the two in setup text is this repo's docs-drift class. -- The `/workflow-init` preflight is **out of scope** for this story; a follow-up entry is - filed in `todos.md`. +- `README.md` `## Setup` documents the variable: its name, Claude Code ≥ 2.1.212, that it + must be set in the environment Claude Code is launched from (restarting an already-running + session), that `0` disables auto-backgrounding, that a positive value must exceed the + longest expected gate call, and what goes wrong without it. +- `CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` is **not** documented: it exists in the 2.1.220 + binary's environment string table but its behaviour was never exercised, and naming the + wrong one of the two in setup text is this repo's docs-drift class. +**The shipped surface this change falsifies is larger than the hook**, and is enumerated +rather than gestured at, because a sentence left behind teaches the old mechanism. Three +groups: + +- **Statements that the hook never inspects results.** The inline CLAUDE template scaffolded + by `/workflow-init`, and this repo's own `CLAUDE.md` §5, describe the gate as keying on tool + name alone and treat a pinned-server failure as counting. Both are edited to describe result + classification — while **keeping** the instruction to discount artifact-validation failures, + which classification does not cover and which stays instruction-backed. +- **Statements that opt-out leaves "accurate" counters.** `README.md`, the reminder text in + `codex-gate.sh`, and `commands/workflow-init.md` each say this; all three take §5.2's + contract instead — same semantics as gate-on, never evidence a review happened. +- **Every mapping instruction, not one example.** `commands/workflow-init.md` carries several + independent ones, including a preflight remedy that renames the server *away* from `codex`, + producing exactly the unreachable `mcp____*` configuration decision 1 describes; it + is removed unless it also re-registers the effective server as `codex`. With it: the + `codex-gate.tools` row in `README.md` and the unknown-tool message in `codex-gate.sh`. Each + states the namespace boundary and the register-as-`codex` remedy. + +Invariant 8 keeps `/workflow-init`'s templates inline, so all its occurrences are edited in +place. The hook message and the scaffolded CLAUDE text are prompts, so both carry golden +assertions (§6). +- The `/workflow-init` preflight is out of scope; a follow-up entry is filed in `todos.md`. - Plugin manifest version bump (invariant 12) and a `CHANGELOG.md` entry. ## 10. What this does not do -- It does not make a counted pass mean Codex read the reviewed bytes. Gate-B validity stays - content-derived; this change only stops calls that returned no review from counting. -- It does not verify third-party envelopes. Those count uninspected, with disclosure. -- It does not survive a reworded backgrounding notice on a machine without the variable set - (§4). +Stated at the precision the mechanism supports: + +- It suppresses exactly the classes the §3.3 table marks non-counting, and nothing more. It + does not establish that a counted call reviewed anything: a call the table classifies + `success` whose findings file is missing or malformed still counts, and remains an incomplete pass under the instruction-backed rule in + CLAUDE.md §5. +- **Third-party envelopes are trusted for counting when they match the polarity grammar.** + A mapped tool whose result begins with `success: true` is classified `success`, counted, + and produces **no disclosure** — there is no provenance check and no verification that the + tool reviewed anything. Only envelopes that match *no* anchor count with disclosure. This + is the trust the design places in decision 1, stated rather than implied. +- It does not survive a reworded backgrounding notice; §4 defines that residual and the + runtimes it covers, and is the only place that does. - It does not classify a post-dispatch cancellation, which was never produced (§8). +- It does not serialize counter mutation, nor defend against a hostile `.context/` (§5.1) — + both pre-existing, both filed. +- Gate-B validity stays content-derived; nothing here changes what the fingerprint compares. + +## 11. Implementation contracts deferred to the plan + +Raised as MAJOR at Gate-A pass 3 and **dispositioned, not resolved**: each is a real +requirement whose natural form is executable shell plus a test, not prose. They are listed +here because the plan must carry them explicitly at its top — a deferred obligation that +lives only in a review artifact is one nobody inherits. The plan's own Gate A pins each +against real code. + +1. **The `jq`-free scanner as a state machine.** Quote state, consecutive-backslash parity, + value boundaries, and what "depth 1" means operationally. §3.1 states the *contract* + (structural with `jq`; conservative and non-guessing without); the plan states the + machine. +2. **Duplicate and malformed key cases.** Which candidate wins when a depth-1 key repeats, + and what happens on malformed JSON. §3.1 sends ambiguity to `unrecognized`; the plan + defines what counts as ambiguous. **Which block is selected is NOT deferred** — §3.1 + settles it as the first array element whose `type` is `text`, and the plan defines only + how the `jq`-free scanner locates that already-settled block. +3. **Accepted raw encodings for JSON whitespace.** Captured inner newlines are the two bytes + `\n`, which POSIX `[[:space:]]` does not match. Every accepted encoding around each prefix + token is enumerated in the plan, with compact, tab and CRLF fixtures. +4. **The full notice grammar.** §4 fixes the anchor; the plan fixes the variable spans — + duration format, task-id boundary, quote representation — and the near-miss cases that + must *not* match. +5. **The complete marker state table.** §5.2 covers the disclosure lifecycle; the plan + extends it to the Cartesian states across both disclosure markers and `bgAdvice`, + including coexistence precedence and every write, delete and retry failure. +6. **Composition against every existing emit branch.** §6 defines composition for a pending + disclosure plus a per-occurrence message; the plan covers the Gate-A, Gate-B, WIP, + docs-only and unknown-tool branches, and events that would otherwise emit nothing. +7. **Separator and encoding rules for composed messages.** Exact control-character-safe + separators, wording that distinguishes the prior event from the current one, and the + `jq`-free escaping required to keep the combined document valid. diff --git a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md index 9d3249a..f72948b 100644 --- a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md +++ b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md @@ -60,24 +60,54 @@ whoever reads it later. ## 2. Desired outcome -A Codex call that did not deliver a review does not leave a pass behind. The hook's -recorded state — counters and the Gate-B fingerprint — reflects only calls whose result is -present and reports success. Where a call's outcome cannot reach the hook at all, the +A Codex call whose result is recognizable as not having delivered a review does not leave a +pass behind. The hook's +recorded state — counters and the Gate-B fingerprint — excludes calls whose result reports +failure, calls from which no result is readable, and calls carrying the **recognized** +backgrounding notice. A result this hook cannot recognize is counted, and its disclosure is +attempted — best-effort, not guaranteed: where both the message and its retry record fail to +persist, such a pass is counted silently. + +*(This paragraph is a **summary**. The normative definitions live in the design — the five +classes in its §3.3, the backgrounding residual in its §4, delivery guarantees in its §6. If +they disagree, the design is right and this paragraph is the thing to amend; it is written +in one place so there is one target.)* +(Amended twice, recorded rather than silently rewritten since a reader may have acted on +the earlier wording. **Pass 2:** the original sentence said the state reflects "only calls +whose result is present and reports success", which the fail-open decision for unrecognized +envelopes made false. **Pass 5:** the replacement said the state excludes "calls whose +outcome never reached the hook" — but a backgrounding notice whose wording has changed is +exactly such a call, and the design counts it as unrecognized, so the exclusion is now +scoped to the *recognized* notice and the exception is stated outright.) Where a call's +outcome cannot reach the hook at all, the operator learns that from the reminder, together with the setting that prevents it, instead of receiving a silent ✓. The workflow stays usable on machines that lack that setting: a discarded pass reads as an actionable setup gap, not as a failed review. ## 3. Acceptance criteria -- [ ] Given a `PostToolUse` payload for a Codex gate tool whose result reports failure, no - pass counter advances and no Gate-B fingerprint is stored — checked against both - payloads captured on 2026-07-30 (`CODEX_EXECUTION_FAILED`, `CODEX_TIMEOUT`). -- [ ] Given a payload carrying no readable Codex result at all — the auto-backgrounding - notice being the observed instance — no counter advances and no fingerprint is - stored. +- [ ] Both payloads captured on 2026-07-30 (`CODEX_EXECUTION_FAILED`, `CODEX_TIMEOUT`) + classify as `failure`, so no pass counter advances and no Gate-B fingerprint is stored. + (Amended 2026-07-31 at the design's Gate-A pass 8. This originally read "whose result + reports failure", which is broader than what the design recognizes: an envelope that + reports failure but has been *reordered* is `unrecognized`, and counts by the fail-open + decision. The criterion now names the captured payloads and defers recognition to the + design's §3.3 table, which is the single normative definition.) +- [ ] Given the recognized auto-backgrounding notice, no counter advances and no fingerprint + is stored. - [ ] In that case the reminder states that the pass was discarded because the call left the foreground, distinguishes this from a failed review, and names `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` as the fix. +- [ ] Given a payload from which no result text can be obtained, no counter advances and no + fingerprint is stored, and the reminder names the causes that actually apply — a + hooks-API payload contract change, or a mapped tool returning empty or non-text + content — rather than diagnosing backgrounding. + (Amended 2026-07-31, during the design's Gate-A pass 4. The first two criteria + originally read as one: "a payload carrying no readable Codex result at all — the + auto-backgrounding notice being the observed instance". That was true while those were + one case; the design separated them into `backgrounded` and `no-result`, which have + different causes and need different diagnoses, and the original wording would have + required reporting a third-party tool's empty response as a backgrounded call.) - [ ] A Codex call whose result reports success still counts exactly as it does today: the counter advances and, for Gate B, the fingerprint is stored. - [ ] Behaviour on all of the above is identical whether or not `jq` is on `PATH`. diff --git a/todos.md b/todos.md index 7be7c90..fd0a984 100644 --- a/todos.md +++ b/todos.md @@ -132,6 +132,26 @@ driven by recurrence rather than by enthusiasm. confusing one-off red: it is a step a human must know about and repeat, on a command AGENTS.md presents as "what CI runs". Two occurrences of the same class now; counts toward whatever trigger this row is eventually escalated on. +- [ ] **Gate-pass counters are read-modify-write, with no serialization.** `bump_count` + reads, increments and writes; two `PostToolUse` events arriving concurrently can lose + an increment, and the fresh-count, fingerprint and pass-count writes are independent, + so a reader can observe a mixed snapshot. This **predates** the result-classification + change and applies to every counter — that change adds files with the same property, + not new exposure, which is why it was documented as a stated contract there rather + than fixed asymmetrically. Raised as MAJOR at Gate-A pass 2 of that design under the + risk lens. *Trigger: the first observed lost increment, or when batch/orchestrator + work makes concurrent gate calls real* — the parked story for that is where + concurrency stops being hypothetical. Until then: known, stated, unexploited. +- [ ] **The hook trusts `.context/` and does not reject non-regular state targets.** A + globally installed hook creates and truncates files inside a repository-controlled + directory, following symlinks; a hostile workspace could point a state file or marker + at another user-writable path. **Pre-existing** for every state file the hook already + writes — the result-classification change adds markers with identical properties, so + fixing only the new ones would be inconsistent, and fixing all of them was out of that + story's scope. Raised as MAJOR (medium confidence) at Gate-A pass 2 under the security + lens. Any fix must keep invariant 1 (always exit 0) on the rejection path. + *Trigger: the first security-`high` story touching the hook, or a real report of a + hostile-workspace scenario.* - [ ] **Temp-index writes land in the real object database.** `git add -A` against the throwaway index writes loose blobs/trees into the user's repo (verified: 3 → 5 objects per review). Unreachable, so gc collects them, but a temporary From 810503c67581f5c575d7993bb59fe6094b3331b5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sat, 1 Aug 2026 11:13:10 +0200 Subject: [PATCH 05/10] docs(todos): each gate cycle destroys the prior cycle's review record Slot names carry no cycle-unique component, and the section 5 protocol mandates deleting every target before each call -- correctly, since a surviving prior file is indistinguishable from a fresh one. So a second cycle in the same repo silently erases the first cycle's findings artifacts. Observed rather than theorised: this cycle's Gate-A pass-1 call deleted the 2026-07-26 profiles cycle's gate-a-spec-pass-1.md, which is unrecoverable because .context is git-ignored. Section 5 covers concurrent calls racing on one slot; it does not cover sequential cycles reusing them. Gate B skipped: behaviourally trivial (one backlog row; no executable path changes) and the change cites no profiled story, so it keeps the pre-existing judgement-based skip. Invariant checks green. --- todos.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/todos.md b/todos.md index fd0a984..55174bf 100644 --- a/todos.md +++ b/todos.md @@ -256,6 +256,19 @@ backlog. editable prose should reword instead.) And the ledger itself had to be **excluded** from both checks, because a ledger that quotes defects self-rejects the checks that detect them. +- [ ] **Each Gate cycle destroys the previous cycle's review record.** Slot names + (`gate-a-spec-pass-

`, `gate-b--pass-

`) carry no cycle-unique component, + and §5 mandates deleting every target file before each call — correctly, since a + surviving prior file is indistinguishable from a fresh one. The consequence is that a + second cycle in the same repo silently erases the first cycle's findings artifacts. + **Observed, not theorised:** the result-classification cycle's pass-1 call deleted the + 2026-07-26 profiles cycle's 11 KB `gate-a-spec-pass-1.md`. `.context/` is git-ignored, + so it is unrecoverable. §5 anticipates *concurrent* calls racing on one slot and says + so; it does not cover *sequential cycles* reusing them. Note the dispositions and + resume-note companions have the same property. Any fix has to keep the pre-call delete + — that check is load-bearing — so it is about naming (a cycle component in the slot) or + archiving, not about relaxing the protocol. *Trigger: the next round touching the §5 + file protocol.* - [ ] **Finding B — a §5 version stamp, so a scaffolded CLAUDE.md can tell it lags the installed plugin.** Split out of the canvas-findings round after two Gate-A passes showed it is a design, not a sentence. Spec questions: a semantic §5 locator From 13c2a448db64cda943d805c0212e2ea6226ab120 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sat, 1 Aug 2026 11:17:49 +0200 Subject: [PATCH 06/10] docs(plan): implementation plan for gate-pass result classification Eleven tasks, TDD steps throughout. Opens with the carried-obligations checklist from the Gate-A closure record: seven deferred implementation contracts (A1-A7), the shipped-doc scope discovered at spec pass 8 (B1-B3), and three accepted residuals that must survive unchanged (C1-C3). Each names the task that discharges it. Ordering is load-bearing: fixtures and test helpers land before any behaviour change, so the suite is green on both sides of the classifier; emit's status change lands before the markers that depend on it; the shipped-doc edits land after the behaviour they describe is real. Carries the watch-item forward: if this plan's Gate A concentrates on A5-A7, stop and surface rather than elaborate -- a simpler composition semantics is the named pressure valve and that trade is decided upstream. Gate B N/A: docs-only. Gate A on this plan runs next, per CLAUDE.md section 5. --- ...6-08-01-gate-pass-result-classification.md | 826 ++++++++++++++++++ 1 file changed, 826 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md diff --git a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md new file mode 100644 index 0000000..0e3d550 --- /dev/null +++ b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md @@ -0,0 +1,826 @@ +# Gate-Pass Result Classification Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop the gate hook from counting Codex calls that delivered no review, by classifying the tool result before touching any pass state. + +**Architecture:** One classifier runs in the `PostToolUse` branch for the two gate tools. It locates the result text block (structurally with `jq`, via an `awk` scanner without), produces one shared escaped representation, and returns one of five classes. Three classes write no gate-pass state; two behave exactly as today. Diagnostic markers are separate from pass state and are always best-effort. + +**Tech Stack:** POSIX `sh`, POSIX `awk`, optional `jq`. No new dependencies. + +**Spec:** `docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md` +**Story:** `docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md` — read its header for the live profile at each gate pass; never copy the values here. +**Gate-A closure:** `.context/codex-reviews/gate-a-spec-CLOSURE.md` + +--- + +## Carried obligations — check off or explicitly re-disposition during this plan's Gate A + +Nothing here may silently evaporate. Each item names the task that discharges it. + +### A. Implementation contracts deferred from spec §11 + +- [ ] **A1** — The `jq`-free scanner as a state machine: quote state, backslash parity, value boundaries, the operational meaning of "depth 1". → Task 4 +- [ ] **A2** — Recognition of duplicate depth-1 keys and of malformed JSON (the *classes* are settled in the spec; only recognition is deferred). → Task 4 +- [ ] **A3** — Accepted raw encodings around every token, including the blank-byte grammar and its ordering against canonical-form validation. → Task 5 +- [ ] **A4** — Full backgrounding-notice grammar: duration format, task-id boundary, quote representation, and the near-misses that must NOT match. → Task 7 +- [ ] **A5** — Complete marker state table across both disclosure markers and `bgAdvice`, with coexistence precedence and every write/delete/retry failure. → Task 8 +- [ ] **A6** — Composition against every existing emit branch (Gate A, Gate B, WIP, docs-only, unknown-tool) and events that would otherwise emit nothing. → Task 9 +- [ ] **A7** — Separator and encoding rules for composed messages. → Task 9 + +### B. Shipped-doc scope from Gate-A pass 8 — SCOPE, NOT WORK + +Determines which files the change touches. **None is edited before Task 10.** + +- [ ] **B1** — The inline CLAUDE template in `commands/workflow-init.md`, and this repo's `CLAUDE.md` §5, both state the hook keys on tool name and never inspects results. +- [ ] **B2** — "Accurate counters" on opt-out appears in `README.md`, the `codex-gate.sh` reminder text, and `commands/workflow-init.md`. +- [ ] **B3** — Every mapping instruction in `commands/workflow-init.md` — including a preflight remedy that renames the server *away* from `codex` — plus the `README.md` knob row and the unknown-tool hook message. + +### C. Accepted residuals — must survive unchanged, not be engineered away + +- [ ] **C1** — Spec §4: a reworded backgrounding notice, on any runtime where auto-backgrounding is still effective, is counted again. +- [ ] **C2** — Spec §5.2: an `unrecognized` call whose disclosure is neither delivered nor persisted is counted silently. +- [ ] **C3** — Spec §5.1: counter mutation is unserialized and `.context/` is trusted. Both pre-existing, both filed in `todos.md`. + +### Watch-item + +**If this plan's Gate-A findings concentrate on A5–A7, STOP AND SURFACE rather than elaborating.** Those three grew by accretion across the spec's own passes. The named pressure valve is a *simpler composition semantics* — dropping compose-into-one-emit for a single deferred flush, or accepting a duplicated disclosure instead of tracking pending/shown separately. That trade changes what the design promises about delivery, so it is decided upstream by the human. + +**A genuine `NO FINDINGS` exit is expected here.** If this loop also ends on judgement, stop and surface: two judgement exits in one cycle is a pattern. + +--- + +## Global Constraints + +- **The hook always exits 0.** Every path, including a classifier failure, an unwritable marker and a failed emit. Withholding a *count* must never become a non-zero *exit*. +- **POSIX `sh` only.** No bash-isms. `shellcheck --shell=sh` is in the battery. POSIX `awk` is permitted (already POSIX; no new dependency). +- **`jq` is optional.** With and without it, every unambiguous payload must reach the same class. Where they cannot, the payload is `unrecognized` — never a guess. +- **Loose in the firing direction.** On uncertainty, fire. A false ✓ is the dangerous direction. +- **Hook messages are prompts.** `docs/prompt-standards.md`, all 12 items, for every string added or changed. +- **Every plugin change requires a manifest version bump** (invariant 12) — Task 11. +- **Quality battery:** the `quality` row of `AGENTS.md` § Commands. Run it before every commit. +- **Never `git add -A`.** Stage the exact paths named in each task. + +## File Structure + +| File | Responsibility | Task | +|---|---|---| +| `plugins/dev-workflow/hooks/fixtures/*.json` | Sanitized captured payloads; the suite's only source of payload shapes | 1 | +| `plugins/dev-workflow/hooks/fixtures/README.md` | Provenance and the fact of sanitization | 1 | +| `plugins/dev-workflow/hooks/codex-gate.test.sh` | Helpers updated first, then coverage per task | 2, and every task after | +| `plugins/dev-workflow/hooks/codex-gate.sh` | `emit` status, locator, classifier, state effects, messages | 3–9 | +| `README.md`, `CLAUDE.md`, `commands/workflow-init.md` | The shipped statements this change falsifies | 10 | +| `plugins/dev-workflow/.claude-plugin/plugin.json`, `CHANGELOG.md` | Version bump and entry | 11 | + +**Ordering rationale:** fixtures and helpers land before any behaviour change, so the suite is green on both sides of the classifier. `emit`'s status change lands before the markers that depend on it. Documentation lands after the behaviour it describes is real. + +--- + +### Task 1: Fixture home + +**Files:** +- Create: `plugins/dev-workflow/hooks/fixtures/{shape0-success,shape1-fast-fail,shape2-executor-timeout,shape3-backgrounding-notice}.json` +- Create: `plugins/dev-workflow/hooks/fixtures/README.md` +- Source (untracked, do not ship): `.context/probe-payloads/` + +**Interfaces:** +- Produces: `$FIXTURES` resolution rule used by every later task — `FIXTURES="$(dirname "$0")/fixtures"`. + +- [ ] **Step 1: Sanitize and copy the four existing captures** + +`tool_response` must stay byte-exact; only machine-specific metadata is neutralized. + +```sh +cd /Users/daniel/DEVELOPMENT/APPS/dev-workflow-kit +mkdir -p plugins/dev-workflow/hooks/fixtures +for pair in \ + "shape0-success.json:shape0-success.json" \ + "shape1-fast-fail-execution-failed.json:shape1-fast-fail.json" \ + "shape2-executor-timeout.json:shape2-executor-timeout.json" \ + "shape3-backgrounding-notice.json:shape3-backgrounding-notice.json" +do + src=".context/probe-payloads/${pair%%:*}" + dst="plugins/dev-workflow/hooks/fixtures/${pair##*:}" + jq '.session_id="00000000-0000-0000-0000-000000000000" + | .transcript_path="/dev/null" + | .cwd="/tmp/fixture-repo" + | .tool_use_id="toolu_fixture" + | .prompt_id="00000000-0000-0000-0000-000000000000"' "$src" > "$dst" +done +``` + +- [ ] **Step 2: Verify `tool_response` survived byte-exact** + +```sh +for pair in \ + "shape0-success.json:shape0-success.json" \ + "shape1-fast-fail-execution-failed.json:shape1-fast-fail.json" \ + "shape2-executor-timeout.json:shape2-executor-timeout.json" \ + "shape3-backgrounding-notice.json:shape3-backgrounding-notice.json" +do + a=$(jq -c '.tool_response' ".context/probe-payloads/${pair%%:*}") + b=$(jq -c '.tool_response' "plugins/dev-workflow/hooks/fixtures/${pair##*:}") + [ "$a" = "$b" ] && echo "OK ${pair##*:}" || { echo "DRIFT ${pair##*:}"; exit 1; } +done +``` + +Expected: four `OK` lines. Any `DRIFT` means the sanitizer touched the object under test — stop and fix the filter. + +- [ ] **Step 3: Capture the review-tool fixture against a disposable synthetic repo** + +`tool_response` cannot be redacted afterwards, so it must never contain real work. Create a throwaway repo with invented content, run one `mcp__codex__review` against it, and capture the payload with the same hook-instrumentation method recorded in `.context/probe-payloads/INDEX.md`. Save as `shape0-success-review.json`, sanitized by the Step 1 filter. + +**Then read the whole fixture end to end before staging it.** Confirm no real path, code excerpt or finding text appears. This is a manual gate; it has no automated check because the thing it guards against is content, not shape. + +- [ ] **Step 4: Write the provenance README** + +```markdown +# Hook test fixtures + +Real `PostToolUse` payloads captured from the hook, used by `codex-gate.test.sh`. + +**Sanitized, and how:** `session_id`, `transcript_path`, `cwd`, `tool_use_id` and +`prompt_id` are replaced with fixed placeholders. **`tool_response` is byte-exact** — +it is the object under test, so it is never rewritten. + +`shape0-success-review.json` was captured against a disposable synthetic repository +with invented content, because a real review response embeds file paths, code and +findings that byte-exactness would preserve. + +| File | Class it exercises | +|---|---| +| `shape0-success.json` | `success` (exec) | +| `shape0-success-review.json` | `success` (review) | +| `shape1-fast-fail.json` | `failure` — `CODEX_EXECUTION_FAILED` | +| `shape2-executor-timeout.json` | `failure` — `CODEX_TIMEOUT` | +| `shape3-backgrounding-notice.json` | `backgrounded` | +``` + +- [ ] **Step 5: Run the battery and commit** + +```sh +sh plugins/dev-workflow/hooks/codex-gate.test.sh && sh scripts/check-invariants.sh +git add plugins/dev-workflow/hooks/fixtures +git commit -m "test(hooks): ship sanitized captured payloads as fixtures" +``` + +--- + +### Task 2: Existing helpers carry real results + +Under this design a payload with no `tool_response` is `no-result` and stops counting. Every existing counting call site sends exactly that. They are updated **before** the classifier lands, so the suite is green on both sides. + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.test.sh:33-38` (helpers), plus the direct payloads at lines ~225, ~365, ~369 and the mapped-tool payloads at ~465, ~481 + +**Interfaces:** +- Produces: `rev()`, `execp()`, `codextool()` all carrying a success envelope; `rev_noresult()` for the deliberately result-less case. + +- [ ] **Step 1: Add the fixture path and envelope helpers** + +```sh +FIXTURES="$(dirname "$0")/fixtures" +# The success envelope every "a gate call happened" helper now carries. +succ() { jq -c '.tool_response' "$FIXTURES/shape0-success.json"; } +payload() { # $1 = tool name, $2 = tool_response JSON + printf '{"hook_event_name":"PostToolUse","tool_name":"%s","tool_input":{},"tool_response":%s}' "$1" "$2" +} +``` + +- [ ] **Step 2: Rewrite the helpers to use it** + +```sh +rev() { run "$(payload mcp__codex__review "$(succ)")" >/dev/null; } +execp() { run "$(payload mcp__codex__exec "$(succ)")" >/dev/null; } +codextool() { run "$(payload "$1" "$(succ)")"; } +rev_noresult() { run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}'; } +``` + +`rev_noresult` exists so the `no-result` tests state their case rather than inheriting it. + +- [ ] **Step 3: Replace every direct result-less gate payload** + +Search and convert: + +```sh +grep -n '"tool_name":"mcp__codex__\(exec\|review\)"' plugins/dev-workflow/hooks/codex-gate.test.sh +``` + +Every hit that represents *a gate call that should count* becomes `execp`/`rev`/`payload …`. The only hits left without `tool_response` are inside `rev_noresult`. + +- [ ] **Step 4: Run the suite against the UNCHANGED hook** + +Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh` +Expected: `all passed`. The hook ignores `tool_response` today, so adding it changes nothing — that is the point: this task is behaviour-neutral and provable. + +- [ ] **Step 5: Commit** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "test(hooks): drive gate calls with real result envelopes" +``` + +--- + +### Task 3: `emit` propagates writer status + +Spec §6 defines marker-writing as conditional on "a complete hook JSON document was written". `emit` currently returns 0 unconditionally after its output command. + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — `emit()` +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**Interfaces:** +- Produces: `emit` returns `0` written, `1` suppressed by the off-switch, `2` write failed. Callers treat **only 0** as "shown". + +- [ ] **Step 1: Write the failing test** + +```sh +# Section: emit reports writer failure +reset_all +: > "$off_file_unused" 2>/dev/null || true +out=$(rev 2>/dev/null >&-; echo "rc=$?") # stdout closed +printf '%s' "$out" | grep -q 'rc=0' && pass "hook still exits 0 with stdout closed" \ + || fail "hook still exits 0 with stdout closed" +``` + +- [ ] **Step 2: Run it to see it fail** + +Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -i 'stdout closed'` +Expected: FAIL — nothing distinguishes a failed write yet. + +- [ ] **Step 3: Make `emit` report its writer** + +```sh + [ -f "$off_file" ] && return 1 + if command -v jq >/dev/null 2>&1; then + jq -cn --arg ev "$event" --arg ctx "$1" --arg msg "$2" \ + '{hookSpecificOutput:{hookEventName:$ev,additionalContext:$ctx},systemMessage:$msg}' || return 2 + else + ctx=$(printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g') + msg=$(printf '%s' "$2" | sed 's/\\/\\\\/g; s/"/\\"/g') + printf '{"hookSpecificOutput":{"hookEventName":"%s","additionalContext":"%s"},"systemMessage":"%s"}\n' \ + "$event" "$ctx" "$msg" || return 2 + fi + return 0 +``` + +- [ ] **Step 4: Fix the stale comment in the same edit** + +The `emit` header says state tracking "keeps running so re-enabling is accurate". Spec §5.2 retires that word. Replace with: *"State tracking keeps running while off, so re-enabling carries the same counting semantics as if the gate had been on — not a guarantee that every counted call was reviewed."* This is item **B2**'s hook-side occurrence; the other two sites are Task 10. + +- [ ] **Step 5: Verify and commit** + +Run the battery. Expected: `all passed`, `shellcheck` clean. + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "fix(hooks): emit reports whether it actually wrote" +``` + +--- + +### Task 4: Locate the result block — discharges A1, A2 + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — new `locate_result()` above the `case "$event"` dispatch +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**Interfaces:** +- Produces: `locate_result` prints the located block's text **in escaped form** on stdout and returns `0`; returns `1` for *unambiguously nothing there* (→ `no-result`); returns `2` for *cannot determine* (→ `unrecognized`). Consumed by Task 5. + +**A1 — the scanner state machine, stated operationally.** The scanner walks the payload one character at a time holding three variables: `instr` (inside a JSON string), `esc` (the previous character was an unescaped backslash), and `depth` (object/brace nesting, counted only while `instr` is 0). **Backslash parity** falls out of `esc` toggling rather than counting: each backslash flips it, any other character clears it, and a quote closes the string only when `esc` is 0. **"Depth 1"** means: the key's opening quote occurs while `depth == 1`, i.e. directly inside the payload's single top-level object. + +**A2 — recognition.** A second depth-1 `tool_response` key is *ambiguity*, returning 2. Malformed JSON is **not** classified: if the outer document cannot be walked to a balanced end, the scanner returns 2 and — per spec §3.3 — the hook has already failed to route the event at all, so nothing is emitted. + +- [ ] **Step 1: Write the failing tests, one per contract case** + +```sh +# Section: locate_result +lr() { printf '%s' "$1" | sh "$HOOK_LOCATE"; } # thin harness exposing locate_result + +t='{"tool_input":{"instruction":"see \"tool_response\" docs"},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' +lr "$t" | grep -q '\\"success\\": true' && pass "ignores tool_response quoted in tool_input" \ + || fail "ignores tool_response quoted in tool_input" + +t='{"tool_response":[{"type":"image","data":"x"},{"type":"text","text":"{\"success\": false}"}]}' +lr "$t" | grep -q '\\"success\\": false' && pass "skips a non-text block" || fail "skips a non-text block" + +t='{"tool_response":[{"type":"text","text":"a"}],"tool_response":[{"type":"text","text":"b"}]}' +lr "$t"; [ $? -eq 2 ] && pass "duplicate depth-1 key is ambiguous" || fail "duplicate depth-1 key is ambiguous" + +t='{"tool_response":[]}' +lr "$t"; [ $? -eq 1 ] && pass "empty array is nothing-there" || fail "empty array is nothing-there" + +t='{"tool_response":{"type":"text","text":"x"}}' +lr "$t"; [ $? -eq 1 ] && pass "non-array container is nothing-there" || fail "non-array container is nothing-there" +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -c '^FAIL'` +Expected: 5 failures — `locate_result` does not exist. + +- [ ] **Step 3: Implement the scanner** + +```sh +# Locates the depth-1 "tool_response" value and returns the first text block's +# text in ESCAPED form. Exit 0 = located; 1 = unambiguously nothing there; +# 2 = cannot determine (ambiguous / malformed). +locate_scan() { + printf '%s' "$payload" | awk ' + BEGIN { RS = "\0"; found = 0; dup = 0 } + { + s = $0; n = length(s); depth = 0; instr = 0; esc = 0; keyq = 0 + for (i = 1; i <= n; i++) { + c = substr(s, i, 1) + if (instr) { + if (esc) { esc = 0 } + else if (c == "\\") { esc = 1 } + else if (c == "\"") { instr = 0; if (keyq) { key = buf; keyq = 0 } } + else if (keyq) { buf = buf c } + continue + } + if (c == "\"") { instr = 1; if (depth == 1) { keyq = 1; buf = "" } ; continue } + if (c == "{" || c == "[") { depth++; continue } + if (c == "}" || c == "]") { depth--; continue } + if (c == ":" && depth == 1 && key == "tool_response") { + if (found) { dup = 1; break } + found = 1; vstart = i + 1 + key = "" + } + } + if (dup) { exit 2 } + if (!found) { exit 1 } + print substr(s, vstart) + exit 0 + }' +} +``` + +The printed remainder is handed to a small block-picker that walks the array for the first `"type":"text"` element and prints its raw `"text"` value — still escaped, because nothing decoded it. + +- [ ] **Step 4: Implement the `jq` path with its span check** + +```sh +locate_jq() { + blk=$(printf '%s' "$payload" | jq -r ' + if (.tool_response | type) != "array" then empty + else ( .tool_response[] | select((type == "object") and (.type == "text") and ((.text | type) == "string")) | .text ) + end' 2>/dev/null | head -n1) || return 2 + [ -n "$blk" ] || return 1 + enc=$(printf '%s' "$blk" | jq -Rs . | sed 's/^"//; s/"$//') + # Spec §3.1: exactly one occurrence, and inside the located span. + occ=$(printf '%s' "$payload" | grep -o -F "$enc" | wc -l | tr -d ' ') + [ "$occ" = 1 ] || return 2 + printf '%s' "$enc" +} +``` + +- [ ] **Step 5: Run the tests to verify they pass, in BOTH parser environments** + +```sh +sh plugins/dev-workflow/hooks/codex-gate.test.sh +PATH=/usr/bin:/bin sh -c 'command -v jq >/dev/null && echo "jq present"; sh plugins/dev-workflow/hooks/codex-gate.test.sh' +``` + +Expected: `all passed` in both. Follow the suite's existing `jq`-absent pattern (a stub `jq` earlier on `PATH` that fails) for the second run. + +- [ ] **Step 6: Extraction parity assertion** + +For every fixture, both locators must hand the matcher **byte-identical** input. Assert the input, not the final class — two locating bugs can cancel out in the class. + +```sh +for f in "$FIXTURES"/*.json; do + a=$(payload_from "$f" | with_jq locate_result) + b=$(payload_from "$f" | without_jq locate_result) + [ "$a" = "$b" ] && pass "extraction parity: $(basename "$f")" || fail "extraction parity: $(basename "$f")" +done +``` + +- [ ] **Step 7: Commit** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "feat(hooks): locate the result block in both parser environments" +``` + +--- + +### Task 5: Classify — discharges A3 + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — new `classify()` +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**Interfaces:** +- Consumes: `locate_result` (Task 4). +- Produces: `classify` prints one of `success|failure|backgrounded|no-result|unrecognized`. Consumed by Task 6. + +**A3 — the blank grammar and its ordering.** Blank is defined on the **escaped bytes**: the block is blank when it contains nothing but ASCII space and the two-byte sequences `\n`, `\t`, `\r`. A Unicode-escaped space is *not* blank by this rule and does not need to be — **canonical-form validation runs first** (Task 4, Step 4), so any encoding `jq` would normalize has already returned 2 and become `unrecognized` before the blank test is reached. That ordering is the contract, and Step 3 tests it directly. + +- [ ] **Step 1: Write the failing tests** + +```sh +c() { CLASSIFY_INPUT="$1" sh "$HOOK_CLASSIFY"; } + +[ "$(c "$(cat "$FIXTURES/shape0-success.json")")" = success ] && pass "success fixture" || fail "success fixture" +[ "$(c "$(cat "$FIXTURES/shape1-fast-fail.json")")" = failure ] && pass "fast-fail fixture" || fail "fast-fail fixture" +[ "$(c "$(cat "$FIXTURES/shape2-executor-timeout.json")")" = failure ] && pass "timeout fixture" || fail "timeout fixture" +[ "$(c "$(cat "$FIXTURES/shape3-backgrounding-notice.json")")" = backgrounded ] && pass "notice fixture" || fail "notice fixture" + +# blank grammar +[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":" \n\t "}]')")" = no-result ] \ + && pass "blank text is no-result" || fail "blank text is no-result" +# ordering: unicode-escaped space fails canonical form FIRST +[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":" "}]')")" = unrecognized ] \ + && pass "unicode space is unrecognized, not blank" || fail "unicode space is unrecognized, not blank" +# reordered failure envelope counts as unrecognized, per the table +[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":"{\"status\": \"error\", \"success\": false}"}]')")" = unrecognized ] \ + && pass "reordered envelope is unrecognized" || fail "reordered envelope is unrecognized" +# collision: a success whose summary quotes both literals +[ "$(c "$(cat "$FIXTURES/collision-success-quotes-both.json")")" = success ] \ + && pass "collision fixture classifies by its own envelope" || fail "collision fixture classifies by its own envelope" +``` + +Create `collision-success-quotes-both.json` and `collision-failure-quotes-true.json` in `fixtures/` by editing a copy of `shape0-success.json` / `shape1-fast-fail.json` so the `summary` field contains both marker literals. These two are **synthetic by necessity** — no real call produces them — and the README says so. + +- [ ] **Step 2: Run them to see them fail** + +Expected: 8 failures — `classify` does not exist. + +- [ ] **Step 3: Implement** + +```sh +classify() { + blk=$(locate_result); rc=$? + [ "$rc" = 1 ] && { printf 'no-result'; return; } + [ "$rc" = 2 ] && { printf 'unrecognized'; return; } + case "$blk" in + 'MCP tool "'*'" is still running after '*) printf 'backgrounded'; return ;; + esac + # blank: only ASCII space and the escapes \n \t \r + case "$(printf '%s' "$blk" | sed 's/\\[ntr]//g; s/ //g')" in + '') printf 'no-result'; return ;; + esac + case "$blk" in + '{'*'\"success\":'[[:space:]]*'true'*) printf 'success'; return ;; + '{'*'\"success\":'[[:space:]]*'false'*) printf 'failure'; return ;; + esac + printf 'unrecognized' +} +``` + +The `{` prefix plus immediate key is what enforces "immediately-first"; a reordered envelope falls through. + +- [ ] **Step 4: Run to verify they pass, both parser environments** + +Expected: `all passed` in both. + +- [ ] **Step 5: Commit** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh plugins/dev-workflow/hooks/fixtures +git commit -m "feat(hooks): classify the gate result into five classes" +``` + +--- + +### Task 6: Wire classes to state effects + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh:361-397` (the `PostToolUse` branches) +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**Interfaces:** +- Consumes: `classify` (Task 5). + +- [ ] **Step 1: Write the seeded-preservation tests, both gates** + +Empty-state assertions would pass an implementation that clears earned state, so every case seeds first. + +```sh +for cls in shape1-fast-fail shape2-executor-timeout shape3-backgrounding-notice; do + reset_all; rev; rev # earn two real Gate-B passes + before_count=$(cat "$count"); before_fresh=$(cat "$fresh"); before_state=$(cat "$state") + run "$(payload mcp__codex__review "$(jq -c '.tool_response' "$FIXTURES/$cls.json")")" >/dev/null + [ "$(cat "$count")" = "$before_count" ] && pass "$cls preserves passCount" || fail "$cls preserves passCount" + [ "$(cat "$fresh")" = "$before_fresh" ] && pass "$cls preserves freshCount" || fail "$cls preserves freshCount" + [ "$(cat "$state")" = "$before_state" ] && pass "$cls preserves fingerprint" || fail "$cls preserves fingerprint" + + reset_all; execp; execp # earn two real Gate-A passes + beforeA=$(cat "$countA") + run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/$cls.json")")" >/dev/null + [ "$(cat "$countA")" = "$beforeA" ] && pass "$cls preserves passCountA" || fail "$cls preserves passCountA" +done +``` + +- [ ] **Step 2: Add the success-path discrimination test** + +`success` and `unrecognized` have identical counter and fingerprint effects, so a counter assertion alone cannot tell them apart. + +```sh +reset_all +rev +[ "$(cat "$count")" = 1 ] && pass "success counts" || fail "success counts" +[ ! -f "$unverified_file" ] && pass "success creates no disclosure marker" || fail "success creates no disclosure marker" +``` + +- [ ] **Step 3: Run to see them fail** + +Expected: the preservation tests fail — every class still counts today. + +- [ ] **Step 4: Implement** + +```sh + "$review_tool") + cls=$(classify) + case "$cls" in + success|unrecognized) ;; # fall through to today's behaviour + *) note_discarded "$cls"; exit 0 ;; # no gate-pass state at all + esac + mkdir -p "$state_dir" 2>/dev/null + # ... existing fingerprint/fresh/count logic unchanged ... + ;; + "$exec_tool") + cls=$(classify) + case "$cls" in + success|unrecognized) mkdir -p "$state_dir" 2>/dev/null; bump_count "$countA_file" ;; + *) note_discarded "$cls" ;; + esac + ;; +``` + +- [ ] **Step 5: Run to verify, both parser environments. Commit.** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "feat(hooks): discarded classes write no gate-pass state" +``` + +--- + +### Task 7: `failure`, `no-result` and `backgrounded` messages — discharges A4 + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — `note_discarded()` +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**A4 — the notice grammar.** The anchor is exactly: the block begins `MCP tool "`, and the segment `" is still running after ` occurs before any newline. **Variable spans, explicitly outside the anchor:** the quoted tool name (`codex/exec`, `codex/review`, or any mapped name), the threshold digits and unit, and the task id. **Near-misses that must NOT match:** a block merely containing the phrase later in its text; a result whose summary quotes the whole notice; a block beginning `MCP tool "` with no `is still running after` segment. + +- [ ] **Step 1: Write the failing message tests, including near-misses** + +```sh +out=$(run "$(payload mcp__codex__review "$(jq -c '.tool_response' "$FIXTURES/shape1-fast-fail.json")")") +printf '%s' "$out" | grep -q 'not counted' && pass "failure says not counted" || fail "failure says not counted" + +out=$(rev_noresult) +printf '%s' "$out" | grep -q 'no tool result' && pass "no-result names its cause" || fail "no-result names its cause" +printf '%s' "$out" | grep -q 'mapped tool' && pass "no-result names BOTH causes" || fail "no-result names BOTH causes" + +# near-miss: notice text quoted inside a real envelope must NOT be backgrounded +nm='[{"type":"text","text":"{\"success\": false, \"summary\": \"MCP tool \\\"codex/exec\\\" is still running after 120s\"}"}]' +[ "$(c "$(payload mcp__codex__exec "$nm")")" = failure ] && pass "quoted notice is not backgrounded" || fail "quoted notice is not backgrounded" +``` + +- [ ] **Step 2: Run to see them fail. Step 3: Implement `note_discarded`.** + +```sh +note_discarded() { + case "$1" in + failure) + emit "This Codex call reported failure, so it was not counted as a gate pass and no review fingerprint was stored. An incomplete pass does not count toward the floor." \ + "⚠ Codex call failed — not counted as a gate pass" ;; + no-result) + emit "The payload carried no tool result, so this call was not counted as a gate pass. Two causes are possible and the tool name alone cannot tell them apart: a hooks-API payload contract change (check your Claude Code version and report it — the pinned server cannot produce this shape), or a mapped tool returning empty or non-text content. Check .context/codex-gate.tools for an active mapping and .mcp.json for the effective server." \ + "⚠ Gate call returned no readable result — not counted" ;; + backgrounded) note_backgrounded ;; + esac +} +``` + +- [ ] **Step 4: Implement `note_backgrounded` with the one-shot advice** + +```sh +note_backgrounded() { + if [ -f "$bg_advice_file" ]; then + emit "Gate pass discarded — the call was backgrounded and its result never reached the hook. Not counted. See CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS." \ + "⚠ Gate pass discarded (backgrounded) — not counted" + else + if emit "This gate pass was discarded, not counted: the call was moved to the background at the auto-background threshold (120 s by default), so its result never reached this hook. This is a setup gap, not a failed review." \ + "⚠ Gate pass discarded: set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from, then restart it (it is read at process start, so exporting it inside a tool shell has no effect). 0 disables auto-backgrounding; a positive value must exceed your longest gate call. Requires Claude Code >= 2.1.212."; then + mkdir -p "$state_dir" 2>/dev/null + { : > "$bg_advice_file"; } 2>/dev/null || true + fi + fi +} +``` + +The marker is written **only** on `emit` returning 0 — suppressed or failed writes leave the one-shot unspent. + +- [ ] **Step 5: Test the one-shot and its non-burning. Step 6: Commit.** + +```sh +reset_all; run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/shape3-backgrounding-notice.json")")" >/dev/null +: > "$off_file" +out=$(run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/shape3-backgrounding-notice.json")")") +[ -z "$out" ] && pass "off suppresses the short form" || fail "off suppresses the short form" +``` + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "feat(hooks): report discarded passes with cause and fix" +``` + +--- + +### Task 8: The `unrecognized` disclosure and pending state — discharges A5 + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**A5 — the complete marker table.** Three files: `codex-gate.bgAdvice`, `codex-gate.unverified` (shown), `codex-gate.unverifiedPending` (owed). They are independent; `bgAdvice` has no interaction with the other two, which is why its lifecycle is a single row rather than a matrix. + +| `unverified` | `unverifiedPending` | Event | Result | +|---|---|---|---| +| absent | absent | `unrecognized`, emit returns 0 | write `unverified` | +| absent | absent | `unrecognized`, emit returns 1 (off) | write `unverifiedPending` | +| absent | absent | `unrecognized`, emit returns 2 (write failed) | write `unverifiedPending` | +| absent | present | any unsuppressed event | emit disclosure; on 0 → write `unverified`, delete pending; else retain pending | +| absent | present | `unverified` write fails after successful emit | **retain pending** (duplicate beats loss) | +| present | absent | `unrecognized` again | nothing emitted, nothing written | +| present | present | any | treat as shown; delete pending. Reachable only if a delete failed earlier | +| any | any | marker write fails | proceed, exit 0 — **C2 applies** | + +- [ ] **Step 1: Write the state-transition tests, one per row** + +```sh +reset_all +run "$(payload mcp__codex__review "$(unrec)")" >/dev/null +[ -f "$unverified_file" ] && pass "gate-on unrecognized writes shown" || fail "gate-on unrecognized writes shown" + +reset_all; : > "$off_file" +run "$(payload mcp__codex__review "$(unrec)")" >/dev/null +[ -f "$pending_file" ] && pass "gate-off unrecognized writes pending" || fail "gate-off unrecognized writes pending" +[ "$(cat "$count")" = 1 ] && pass "unrecognized still counts while off" || fail "unrecognized still counts while off" + +rm -f "$off_file"; out=$(rev) +printf '%s' "$out" | grep -q 'cannot verify' && pass "pending flushes on re-enable" || fail "pending flushes on re-enable" +[ ! -f "$pending_file" ] && pass "flush clears pending" || fail "flush clears pending" +``` + +- [ ] **Step 2: Run to see them fail. Step 3: Implement.** + +```sh +note_unverified() { + [ -f "$unverified_file" ] && return 0 + if emit "$UNVERIFIED_CTX" "$UNVERIFIED_MSG"; then + mkdir -p "$state_dir" 2>/dev/null + if { : > "$unverified_file"; } 2>/dev/null; then + rm -f "$pending_file" 2>/dev/null + fi # else: pending retained on purpose + else + mkdir -p "$state_dir" 2>/dev/null + { : > "$pending_file"; } 2>/dev/null || true + fi +} +``` + +`$UNVERIFIED_CTX` enumerates the causes with their remedies, per spec §6 — a changed pinned-server envelope (check the version in `.mcp.json`), a reworded backgrounding notice (the `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` guidance), a mapped third-party envelope (**no user-side fix**), and a hook parser defect (**no operator fix**) — and states that the list is not exhaustive because `unrecognized` is the terminal class. + +- [ ] **Step 4: Verify, both parser environments. Commit.** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "feat(hooks): disclose uninspected passes once per workspace" +``` + +--- + +### Task 9: Composition — discharges A6, A7 + +**Files:** +- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` +- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` + +**A6 — every emit branch.** A pending disclosure can coincide with: the Gate-B STOP/stale/satisfied reminders, the Gate-A floor reminder, the WIP note, the docs-only note, the unknown-tool note, a `failure`/`no-result`/`backgrounded` note, **and events that emit nothing at all**. The last is why the flush is a wrapper rather than a call inside each branch. + +**A7 — separator and encoding.** The two bodies join with `" — "` (space, em dash, space) in `additionalContext` and `" "` in `systemMessage`. **No newline**, because the `jq`-free emitter escapes only backslash and quote, and a literal newline would produce an invalid JSON document. The disclosure is prefixed `Earlier: ` so a prior-event disclosure cannot read as a statement about the current call. + +- [ ] **Step 1: Write the collision test for a silent event** + +```sh +reset_all; : > "$off_file" +run "$(payload mcp__codex__review "$(unrec)")" >/dev/null # owes a disclosure +rm -f "$off_file" +out=$(run '{"hook_event_name":"PostToolUse","tool_name":"Edit","tool_input":{"file_path":"a.ts"}}') +printf '%s' "$out" | grep -q 'Earlier:' && pass "silent event still flushes pending" || fail "silent event still flushes pending" +printf '%s' "$out" | jq -e . >/dev/null 2>&1 && pass "flushed output is one valid JSON document" || fail "flushed output is one valid JSON document" +``` + +- [ ] **Step 2: Run to see it fail. Step 3: Implement the single-emit wrapper.** + +Route every emit through one function that prepends a pending disclosure when one is owed, and — for an invocation that would otherwise emit nothing — flushes it alone before the hook exits. + +- [ ] **Step 4: Assert exactly one JSON document per invocation across every branch** + +```sh +for scenario in gateb_stop gatea_floor wip docsonly unknowntool failure noresult backgrounded silent; do + out=$(run_scenario "$scenario") + [ "$(printf '%s' "$out" | grep -c '^{')" -le 1 ] && pass "$scenario emits at most one document" || fail "$scenario emits at most one document" +done +``` + +- [ ] **Step 5: Commit** + +```sh +git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "feat(hooks): compose an owed disclosure into a single emit" +``` + +--- + +### Task 10: The shipped statements this change falsifies — discharges B1, B2, B3 + +**Files:** +- Modify: `CLAUDE.md` §5; `README.md`; `plugins/dev-workflow/commands/workflow-init.md` + +**This task is where B becomes work.** Nothing in it is edited before this point. + +- [ ] **Step 1: Find every occurrence rather than trusting the list** + +```sh +grep -rn "keys on tool name\|never inspects\|accurate" --include='*.md' . | grep -v 'source-files/\|docs/superpowers/' +grep -rn "execTool\|codex-gate.tools" --include='*.md' . | grep -v 'source-files/\|docs/superpowers/' +``` + +The pass-8 list is a floor, not a census — invariant: the greps decide. + +- [ ] **Step 2: B1 — describe classification, keep the artifact-validation instruction** + +In `CLAUDE.md` §5 and the inline CLAUDE template: replace "the hook counts on tool name and never inspects the result" with what the classifier does. **Keep** the rule that an incomplete pass is discounted regardless of the counter — classification does not cover a `success` call whose findings file is missing, and that remains instruction-backed. + +- [ ] **Step 3: B2 — retire "accurate"** + +Three sites take spec §5.2's contract: opt-out preserves the same counting semantics as gate-on, and the counters are never evidence a review happened. + +- [ ] **Step 4: B3 — every mapping instruction** + +Each states that a mapped name must lie in `mcp__codex__*` and that the remedy is registering the server as `codex`. **Remove the preflight remedy that renames the server away from `codex`** unless it also re-registers the effective server there — as written it produces exactly the unreachable configuration. + +- [ ] **Step 5: Golden assertions for the two prompts** + +The unknown-tool hook message and the scaffolded CLAUDE text are prompts. Pin both with exact-match assertions, following the suite's existing `matches exactly` tests. + +- [ ] **Step 6: All 12 prompt-standards items** + +Review every string added or changed in Tasks 7–10 against `docs/prompt-standards.md` — all 12, not item 10 alone. + +- [ ] **Step 7: Battery and commit** + +```sh +git add CLAUDE.md README.md plugins/dev-workflow/commands/workflow-init.md plugins/dev-workflow/hooks/codex-gate.test.sh +git commit -m "docs: describe result classification where the old mechanism was taught" +``` + +--- + +### Task 11: Version, changelog, and the named verification + +**Files:** +- Modify: `plugins/dev-workflow/.claude-plugin/plugin.json`, `plugins/dev-workflow/CHANGELOG.md` + +- [ ] **Step 1: Bump the manifest version** (invariant 12 — a plugin change without one fails CI). + +- [ ] **Step 2: CHANGELOG entry**, newest first, naming the behaviour change and the three accepted residuals **C1–C3** by name. + +- [ ] **Step 3: The `+check` counterfactual** + +Run the `failure`-class tests against the **pre-change** hook and record that they fail: + +```sh +git stash +sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -E 'FAIL.*(fast-fail|timeout|preserves)' +git stash pop +``` + +This is an observation to record, not an assertion to make. + +- [ ] **Step 4: The named verification** (the story's profile requires it) + +Re-run the probe methodology against the changed hook, recording the counter and fingerprint reading for each: both failure envelopes; `backgrounded` with the variable absent; `backgrounded` prevented with the variable set; one genuine pass that still counts. Method is in `.context/probe-payloads/INDEX.md`. + +- [ ] **Step 5: Full battery, then Gate B** + +Run the `quality` row of `AGENTS.md` § Commands. Then CLAUDE.md §5 Gate B: WIP commit, `mcp__codex__review` against its parent, minimum three passes, findings to file, clean final pass, evidence entry in the closing commit body. + +--- + +## Self-Review + +**Spec coverage:** §3.1 → Task 4; §3.2–3.3 → Task 5; §4 → Task 7 (A4) and C1; §5.1 → Task 6; §5.2 → Task 8 (A5); §6 → Tasks 7–9; §7.1 → Task 1; §7.2 → Task 2; §7.3 → Tasks 4–9; §7.4 → Task 11; §9 → Tasks 10–11; §11 → A1–A7 as mapped. + +**Placeholders:** none. Every code step carries runnable content. Task 10's steps are edits to prose whose exact target text is found by the greps in its Step 1 rather than quoted here, because quoting it would create a fourth copy of the sentences this change exists to correct. + +**Type consistency:** `locate_result` (0/1/2) is consumed only by `classify`; `classify`'s five strings are consumed only by the Task 6 branches; `emit`'s 0/1/2 is consumed by `note_backgrounded` and `note_unverified`. `$FIXTURES`, `payload()`, `succ()` are defined in Tasks 1–2 and used unchanged afterwards. From 6a3cc7dffa6857a8c4652e00469808999d430170 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sat, 1 Aug 2026 11:59:01 +0200 Subject: [PATCH 07/10] docs(todos): note the awk locator as a candidate path for the jq-free parser row The result-classification story builds a POSIX awk locator with proper string-state and backslash-parity handling. Once proven, that machine is the likely reuse path for the escaped-quote defect in input_field's fallback rather than inventing a second escape-aware scanner. Scopes stay separate: that story does not touch input_field. Gate B skipped: behaviourally trivial (a note on an existing backlog row; no executable path changes) and the change cites no profiled story, so it keeps the pre-existing judgement-based skip. Invariant checks green. --- todos.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/todos.md b/todos.md index 55174bf..c934c94 100644 --- a/todos.md +++ b/todos.md @@ -65,7 +65,12 @@ driven by recurrence rather than by enthusiasm. *Trigger: this session's discovery — already fired.* Deliberately not fixed in PR #9, whose scope guard is prompts and templates only; this needs hook code and regression tests. -- [ ] **jq-free parser stops at an escaped JSON quote.** A payload containing +- [ ] **jq-free parser stops at an escaped JSON quote.** *(Candidate path, recorded + 2026-08-01: the result-classification story builds a POSIX awk locator with proper + string-state and backslash-parity handling. Once that exists and is proven against the + captured fixtures, this row's fix can likely reuse it rather than inventing a second + escape-aware scanner. The scopes stay separate — that story does not touch + `input_field` — but whoever takes this row should look there first.)* A payload containing `echo \"quoted\" && git commit -m x` decodes to nothing, so no reminder fires — wrong direction under invariant 2, and only on machines without `jq`. Needs escape-aware decoding or a conservative raw-payload scan, plus tests for escaped From 0dc93d8285bafca548e1b1dace7f1d0ebf7dc52d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sun, 2 Aug 2026 10:13:14 +0200 Subject: [PATCH 08/10] docs(spec,plan): collapse the dual locator and close Gate A at pass 9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec §3.1 is amended to one locator: the jq path's in-span check needed byte offsets jq does not report, so the span came from the scan anyway, leaving jq as a component whose only reachable effect was downgrading an agreed result to unrecognized. The re-encode step, the uniqueness and in-span checks and §7.3's extraction-parity matrix are deleted with it rather than fenced off. The plan is narrowed to contracts, tables, message pairs and per-test oracles; the harness shell moves to Gate B as code that runs and lints. Gate A closed at pass 9 as a pre-authorized judgement exit. The execution notes record all 37 pass-9 findings as named Gate-B obligations, the three fixed at closure because they would have broken execution before Gate B could observe anything, and why nine passes never converged. Gate B: N/A — every staged path is docs/**.md prose. --- ...s-result-classification-execution-notes.md | 130 +++ ...6-08-01-gate-pass-result-classification.md | 1001 +++++++---------- ...iled-codex-call-counts-as-a-pass-design.md | 124 +- 3 files changed, 606 insertions(+), 649 deletions(-) create mode 100644 docs/superpowers/plans/2026-08-01-gate-pass-result-classification-execution-notes.md diff --git a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification-execution-notes.md b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification-execution-notes.md new file mode 100644 index 0000000..89c7c59 --- /dev/null +++ b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification-execution-notes.md @@ -0,0 +1,130 @@ +# Execution notes — gate-pass result classification + +**Gate A closed at pass 9 as a judgement exit, pre-authorized by Daniel** under a rule set +before the number arrived: *clean or Minor-only → close normally; double-digit Blocker+Major, +or any Blocker of the reading class the sweep was meant to exhaust → close there as the +second judgement exit, every open finding dispositioned by name as a Gate-B obligation. No +pass 10 either way.* + +Pass 9 returned **37 findings: 2 Blocker, 31 Major, 4 Minor — 33 Blocker+Major**, and both +Blockers were reading-class. Both clauses fired. + +## Nine passes, and why the count never converged + +| Pass | Blockers | B+M | Total | +|---|---|---|---| +| 1 | 7 | 36 | 40 | +| 2 | 5 | 28 | 32 | +| 3 | 5 | 27 | 35 | +| 4 | 3 | 24 | 29 | +| 5 | 2 | 17 | 19 | +| 6 | 5 | 35 | 37 | +| 7 | 4 | 22 | 26 | +| 8 | 3 | 30 | 33 | +| 9 | 2 | 33 | 37 | + +The watch-item never fired: A5–A7 stayed a minority in every pass, so the instability was +never where the plan predicted it. + +**The plan is not what failed to converge; the review of it did.** Two human course +corrections (collapse the dual locator; narrow what the plan promises) each produced a real +drop, and each was followed by a rebound as the review reached a fresh surface — first the +harness shell, then the product prompts, then the execution procedures. What stayed constant +across passes 5–9 is a defect of *authorship*: a requirement written as prose **beside** an +artifact instead of **into** it, then dispositioned as done. Pass 9 caught three more of +those, one of which (finding 17) was a fix the mechanical sweep itself had claimed. + +**The 2026-08-02 sweep was the right response and was not sufficient.** Thirteen machine +checks found and fixed eight defects that eight read-only passes had missed — including a +rollback that would have byte-verified against the wrong hook. But pass 9 showed two sweep +checks passing *for the wrong reason*: the scratch-clone dry runs defined `$EVIDENCE` and +staged the fix **themselves**, so they proved the `git` mechanics while never testing that +the plan defines either. A check that supplies the thing under test is not a check. +Sweep record: `.context/codex-reviews/gate-a-plan-sweep-2026-08-02.md`. + +## Fixed at closure — three only, and why the line is there + +Everything below is a Gate-B obligation, not a Gate-A fix. Three exceptions were made, +because each would have broken execution *before* Gate B could observe anything: + +- **P9-19 (Blocker)** — `$EVIDENCE` was read by three commit commands and assigned nowhere. + Task 7 gains a **Step 0** defining it, with Steps 2–5 appending to it and a + readable-and-non-empty check before every embed. +- **P9-21 (Blocker)** — the Gate-B amend loop never staged the fix, so a bare `--amend` + would commit nothing new: the next reviewer reads the old range while the fingerprint sees + a changed worktree. The loop now inspects, stages the exact paths, audits the index, + amends, and requires a clean worktree before the next call. +- **P9-17 (Major)** — Task 6's census commands were still recursive `grep`, with prose beside + them saying they should use `git grep`. Commands replaced. *Its stated consequence is not + currently reproducible* — the verbatim commands return **0** `.mcp/` hits today — so the + defect is the prose-not-command gap, not the cache-hit claim. + +Re-verified after those edits: all 8 fenced blocks parse under `sh -n` and pass +`shellcheck --shell=sh` at error level; the ten message assignments parse; drafts 56/56 under +`sh` and `dash`. One of those three fixes introduced a parse error of its own +(`git add ` — `<` is a redirect) which the re-check caught immediately. + +## Gate-B obligations — all 37 pass-9 findings, by name + +Gate B inherits these the way this plan inherited spec §11's deferred contracts. Each must be +confirmed against the real implementation. + +### Contract vs. code — the claim must not outrun `locate.awk` / `match.sh` + +- **P9-2** — A3 says whitespace is accepted at exactly three points; `_token_ends` calls `strip_ws` after the value token, making it four. Enumerate all four sites, their shared alphabet and the 64-unit bound. +- **P9-4** — A2 says "a stray comma" is refused without scoping it to the path walked *before* selection, while the frozen verifier pins `trailing comma AFTER the selected element` as `success`. Qualify it and name that label among the non-refusals. +- **P9-6** — A2 mentions escaped *key* spellings but not that the `type` **value** is compared as raw bytes, so `text` — semantically `text` under spec §3.1 — is skipped and can yield `no-result`. State it, amend the spec, add a fixture, or implement the semantic comparison. +- **P9-32** — `UNVERIFIED_MSG` reduces locator refusals to "oversized or ambiguous", omitting unwalkable structure and the depth cap, and offers no real discriminator. Enumerate the refusal families and label the indistinguishable ones unresolved. +- **P9-35** — the "any `sed` failure maps to `unrecognized`" guarantee ignores jq-free routing, where `field()` also needs `sed`: with `jq` absent and `sed` missing the hook exits before classification. Scope the guarantee to post-routing failure and add the no-route oracle. + +### Dropped or under-specified oracles + +- **P9-1** — the plan claims to contain the five-row class table; it does not. Include it, or name spec §3.3 as the sole table and delete both in-plan claims. +- **P9-3** — only one past-64 fixture (after `{`). Add one per remaining `strip_ws` site. +- **P9-5** — the preceding-non-text duplicate fixture repeats only `type`; add the `text` counterpart. +- **P9-7** — with `.context/plan-drafts/` missing, the reconstruction route points at `verify.sh`, which is in the same missing directory. Track the corpus or inline a non-shell table of all 53 labels and expected classes. +- **P9-11** — `bgAdvice` is called independent but no case covers a pending disclosure composed with background advice when one family's write succeeds and the other's fails. +- **P9-14** — `unrecognized` has no four-combination matrix over default/mapped × exec/review, though the discarded classes do. +- **P9-15** — the review-side `sed`-fault label omits the usable-fingerprint oracle its `awk` counterpart carries. +- **P9-16** — ``, ``, "all nine", "fourteen", "one composed pair" are counts and templates, not the per-test labels the settled scope requires. Enumerate them. +- **P9-18** — Task 6 Step 7 names neither prompt, label, nor independent expected source for its goldens. +- **P9-23** — the named verification demands a per-row expectation and exact row count and supplies neither table. + +### State-machine and table defects + +- **P9-9** — a non-writable `.context` cannot allow creating an absent `unverified` while denying deletion of `pending`; one permission governs both. Needs a selective `rm` shim or another operation-specific fault. +- **P9-10** — the A5 row "any unsuppressed event" includes flush status 1, which *is* suppression, and no row states gate-off retention. Split or rename. +- **P9-12** — `flush_notes` runs unconditionally, so pending debt can emit on a payload whose event could not be routed — contradicting spec §3.3. Gate the flush on successful routing; add a pending-plus-unroutable oracle. +- **P9-13** — one generic encoder-failure label can always fail the *first* substitution and never prove the second returns 2. Split it. + +### Procedure and sequencing + +- **P9-8** — Task 1 Steps 3 and 7 must share one shell and one temp locator, but Steps 4–6 intervene. Move both byte checks after Step 6. +- **P9-20** — `$BASE` must outlive shells, but no durable-note format or checked reload command is defined. +- **P9-22** — Step 4's CI evidence must bind the reviewed tree, yet it precedes the squash that creates it. +- **P9-26** — the squash's log/path/status reads do not bind the tip: a commit created between the reads and `reset --soft` is folded in silently. +- **P9-24** — the non-isolated rollback path activates 0.7.1 in the shared plugin environment with no quiescence requirement, exposing concurrent projects to the known false-checkmark and `dash`-exit behaviour. +- **P9-25** — "the newest commit whose manifest still contains 0.7.1" identifies the last *repository* bytes at that version, not bytes proven *released*; `AGENTS.md` says the version checker cannot establish release, and this repo has a history of unbumped plugin commits. Establish an authoritative release artifact or stop calling them released bytes. + +### Security and privacy + +- **P9-27** — captured `tool_response` blocks contain a real Codex `sessionId`, and the background capture a real task id, while the plan forbids changing any byte inside them and its own disclosure calls session identifiers sensitive. Redact without reserialization and narrow the byte-exact claim, or recapture with inert identifiers. + +### Shipped prompts, against `docs/prompt-standards.md` + +- **P9-28** — `Claude Code gate hook —` names the surface, not the executing model; the claimed source comment recording the prompting-page check does not exist as planned text. (Item 1.) +- **P9-29** — both background prompts require a second backgrounding to be surfaced but give no output format or example. (Item 4.) +- **P9-30** — the four cleanup instructions state the rule without its reason; the stale-artifact rationale lives only in surrounding plan prose. (Item 6.) +- **P9-31** — `FAILURE_CTX` says "every envelope whose first property is success false reaches this state", which outruns the classifier: routing, a located first text block, raw key spelling and the encoding grammar all gate it. (Item 11.) +- **P9-33** — `NORESULT_CTX` tells the model the only external cause is a *mapped* tool while `NORESULT_MSG` correctly allows an effective third-party server under the default `codex` name. The two audiences get contradictory trust-boundary diagnoses. +- **P9-34** — the unknown-tool prompt gains a prefix and paragraphs but never the tagged structure item 5 requires, so Task 6 Step 8's all-12 claim is false for a shipped prompt. +- **P9-36** — the settled §4 anchor does not require a task id, yet both background prompts unconditionally instruct stopping "by the task id in the result". Make it conditional with a no-id path, without narrowing the frozen anchor. +- **P9-37** — `UNVERIFIED_CTX`'s "repeats **only** when its marker cannot be persisted or two hook runs race" omits manual or automated marker deletion. (Item 11.) + +## What Gate B should carry in, beyond these + +The six harness-mechanics obligations already listed in the plan's own +**"Findings that move to Gate B"** section stand unchanged, and are additional to the above. + +**Standing lens, with unusual force:** this change edits `CLAUDE.md` §5 itself. Ask which +existing statements the diff falsifies — including in files it does not touch. diff --git a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md index 0e3d550..ee4397d 100644 --- a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md +++ b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md @@ -4,823 +4,650 @@ **Goal:** Stop the gate hook from counting Codex calls that delivered no review, by classifying the tool result before touching any pass state. -**Architecture:** One classifier runs in the `PostToolUse` branch for the two gate tools. It locates the result text block (structurally with `jq`, via an `awk` scanner without), produces one shared escaped representation, and returns one of five classes. Three classes write no gate-pass state; two behave exactly as today. Diagnostic markers are separate from pass state and are always best-effort. +**Architecture:** One classifier runs in the `PostToolUse` branch for the two gate tools. A single escape-aware `awk` scan locates the result text block and yields its **escaped** bytes; one shell matcher reads those bytes and returns one of five classes. Three classes write no gate-pass state; two behave exactly as today. Output is buffered so an invocation writes at most one hook JSON document. Diagnostic markers are separate from pass state and always best-effort. -**Tech Stack:** POSIX `sh`, POSIX `awk`, optional `jq`. No new dependencies. +**Tech Stack:** POSIX `sh`, POSIX `awk`, POSIX `sed`. `jq` is used only by `emit` and the existing field readers — **never** by classification. **Spec:** `docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md` **Story:** `docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md` — read its header for the live profile at each gate pass; never copy the values here. -**Gate-A closure:** `.context/codex-reviews/gate-a-spec-CLOSURE.md` +**Gate-A closure (spec):** `.context/codex-reviews/gate-a-spec-CLOSURE.md` +**Gate-A closure (this plan) + Gate-B obligations:** `docs/superpowers/plans/2026-08-01-gate-pass-result-classification-execution-notes.md` — **read before executing.** Gate A closed at pass 9 as a pre-authorized judgement exit; 34 open findings are recorded there as Gate-B obligations, and three execution-blocking ones were fixed in this plan at closure. + +--- + +## What this plan specifies, and what it deliberately leaves to execution + +**Narrowed at Gate-A pass 6, human-confirmed 2026-08-01.** Passes 4 and 5 pushed literal test-harness shell into this document; pass 6 then found three skeletons still open *and three fresh defects in the shell added to close the previous ones* — including that the plan's own helper code failed this repo's ShellCheck policy. The document was 1732 lines and converging on a verbatim copy of a test suite that Gate A cannot run, lint, or execute. + +**So the line is drawn here.** This plan specifies: + +- the **contracts** (A1–A7), the **class table**, the **marker state table**, and the accepted **residuals**; +- the **five message pairs**, in full — they are product, they are prompts, and invariant 11 binds them; +- the **behaviour of each task** and its commit boundary; +- for every test, its **label** and its **oracle**: the state it must observe and *what it must fail on*. Coverage stays reviewable without a line of harness shell. + +It does **not** specify the harness shell. That is written test-first during execution and reviewed at **Gate B**, as code that runs and lints. Where a pass-6 finding was about harness mechanics, it is dispositioned **"moves to Gate B"** by name at the end of this document — inherited the way this plan inherited spec §11's deferred contracts, not dropped. + +**The verified code seeds live beside this plan, not inside it.** `.context/plan-drafts/` holds `locate.awk` and `match.sh` — green at **56/56** under `verify.sh`, under both `sh` and `dash` — plus `harness.sh`, a **seed that has never been run** and says so in its own header. They are evidence and starting points; this plan states what they must satisfy. If that directory is missing (it is git-ignored), the bodies must be reconstructed and re-verified against `verify.sh` before use — the cost of keeping them out of the plan, stated rather than discovered. + +**Spec amended earlier in this cycle.** Pass 1 showed the spec's two-locator design was not implementable as written: its in-span check needs byte offsets, which `jq` does not report, so the span had to come from the scan anyway. §3.1 now specifies **one** locator, and the re-encode step, the uniqueness and in-span checks, and §7.3's extraction-parity matrix are **deleted** rather than fenced off. Recorded inline in §3.1 with what it replaced. Human-confirmed. + +**What 56/56 does NOT establish.** `sh` and `dash` invoke the same system `awk`, so it is shell coverage, not `awk`-implementation coverage — and `awk` is load-bearing. Only macOS BWK `awk` exists on the development machine; CI's `ubuntu-*` runs `mawk`. Green in both places is the portability evidence. Until CI has run, portability is **unverified**. --- ## Carried obligations — check off or explicitly re-disposition during this plan's Gate A -Nothing here may silently evaporate. Each item names the task that discharges it. +Nothing here may silently evaporate. Each item names the task that discharges or preserves it. ### A. Implementation contracts deferred from spec §11 -- [ ] **A1** — The `jq`-free scanner as a state machine: quote state, backslash parity, value boundaries, the operational meaning of "depth 1". → Task 4 -- [ ] **A2** — Recognition of duplicate depth-1 keys and of malformed JSON (the *classes* are settled in the spec; only recognition is deferred). → Task 4 -- [ ] **A3** — Accepted raw encodings around every token, including the blank-byte grammar and its ordering against canonical-form validation. → Task 5 -- [ ] **A4** — Full backgrounding-notice grammar: duration format, task-id boundary, quote representation, and the near-misses that must NOT match. → Task 7 -- [ ] **A5** — Complete marker state table across both disclosure markers and `bgAdvice`, with coexistence precedence and every write/delete/retry failure. → Task 8 -- [ ] **A6** — Composition against every existing emit branch (Gate A, Gate B, WIP, docs-only, unknown-tool) and events that would otherwise emit nothing. → Task 9 -- [ ] **A7** — Separator and encoding rules for composed messages. → Task 9 +- [ ] **A1** — The scanner as a state machine: quote state, backslash parity, value boundaries, the operational meaning of "depth 1". → Task 5 +- [ ] **A2** — Recognition of duplicate depth-1 keys and of structures the scan cannot walk. → Task 5 +- [ ] **A3** — Accepted raw encodings around every token, including the blank-byte grammar. → Task 5 +- [ ] **A4** — Full backgrounding-notice grammar: what the anchor fixes, what it leaves variable, and the near-misses that must NOT match. → Task 5 +- [ ] **A5** — Complete marker state table across both disclosure markers and `bgAdvice`. → Task 5 +- [ ] **A6** — Composition against every existing emit branch, and events that would otherwise emit nothing. → Task 4 (mechanism) and Task 5 (the disclosure wired into it) +- [ ] **A7** — Separator and encoding rules for composed messages. → Task 4 -### B. Shipped-doc scope from Gate-A pass 8 — SCOPE, NOT WORK +### B. Shipped-doc scope from the spec's Gate-A pass 8 -Determines which files the change touches. **None is edited before Task 10.** +**Every B edit lands in Task 6, with one stated exception: B2's occurrence inside `codex-gate.sh` is a comment on the function Task 3 rewrites, and is edited there** — separating a comment from the code it describes across four commits is how comments go stale. -- [ ] **B1** — The inline CLAUDE template in `commands/workflow-init.md`, and this repo's `CLAUDE.md` §5, both state the hook keys on tool name and never inspects results. -- [ ] **B2** — "Accurate counters" on opt-out appears in `README.md`, the `codex-gate.sh` reminder text, and `commands/workflow-init.md`. -- [ ] **B3** — Every mapping instruction in `commands/workflow-init.md` — including a preflight remedy that renames the server *away* from `codex` — plus the `README.md` knob row and the unknown-tool hook message. +- [ ] **B1** — `CLAUDE.md:168` and `commands/workflow-init.md:347` say the hook is *"keyed on tool name, and never sees the file"*. **Only the first clause becomes false.** "Never sees the file" is about the **findings file** and stays true. Three further sites say the same thing in different words — Task 6 Step 1. → Task 6 +- [ ] **B2** — "Accurate counters" on opt-out: `README.md:97`, `commands/workflow-init.md:1038`, **and the two suite sites `codex-gate.test.sh:296` and `:306`** → Task 6; `codex-gate.sh:291` → **Task 3**. The suite pair was missing from this list until the pass-8 census caught it: a test whose label asserts the retired claim keeps it true in the one place nobody rereads. +- [ ] **B3** — Mapping instructions: `commands/workflow-init.md:123-124`, `:74` and `:157` (the preflight remedies that rename the server *away* from `codex`), `README.md:98`, and the unknown-tool message in `codex-gate.sh:392`. → Task 6 ### C. Accepted residuals — must survive unchanged, not be engineered away -- [ ] **C1** — Spec §4: a reworded backgrounding notice, on any runtime where auto-backgrounding is still effective, is counted again. -- [ ] **C2** — Spec §5.2: an `unrecognized` call whose disclosure is neither delivered nor persisted is counted silently. -- [ ] **C3** — Spec §5.1: counter mutation is unserialized and `.context/` is trusted. Both pre-existing, both filed in `todos.md`. +- [ ] **C1** — Spec §4: a reworded backgrounding notice, on any runtime where auto-backgrounding is still effective, is counted again. Preserved by Task 5; named in the CHANGELOG (Task 7). +- [ ] **C2** — Spec §5.2: an `unrecognized` call whose disclosure is neither delivered nor persisted is counted silently. **Both directions** — suppressed emit, and failed emit — each combined with a failed pending write. Preserved by Task 5; asserted by two named oracles. +- [ ] **C3** — Spec §5.1: counter mutation is unserialized and `.context/` is trusted. Pre-existing, filed in `todos.md`. Preserved by Task 5 (no locking is added). +- [ ] **C4** — **Concurrent check-emit-write on the diagnostic markers** duplicates or loses a disclosure, in both directions. Spec §6 accepts this under "delivery is best-effort". Recorded rather than repaired — a partial fix over one state family would be the inconsistent repair §5.1 refuses. The sequential tests must not be read as guaranteeing more. Named in the CHANGELOG (Task 7). ### Watch-item -**If this plan's Gate-A findings concentrate on A5–A7, STOP AND SURFACE rather than elaborating.** Those three grew by accretion across the spec's own passes. The named pressure valve is a *simpler composition semantics* — dropping compose-into-one-emit for a single deferred flush, or accepting a duplicated disclosure instead of tracking pending/shown separately. That trade changes what the design promises about delivery, so it is decided upstream by the human. +**If this plan's Gate-A findings concentrate on A5–A7, STOP AND SURFACE rather than elaborating.** The named pressure valve is a *simpler composition semantics* — dropping compose-into-one-emit for a single deferred flush, or accepting a duplicated disclosure instead of tracking pending/shown. That trade changes what the design promises about delivery, so it is decided upstream by the human. It has not fired: A5–A7 drew 6/36, 4/28, ~5/27, ~4/24, 3/17, ~4/35. -**A genuine `NO FINDINGS` exit is expected here.** If this loop also ends on judgement, stop and surface: two judgement exits in one cycle is a pattern. +**A genuine `NO FINDINGS` exit is expected, and narrowing the scope does not change that.** The smaller artifact is a scope correction, not a judgement exit through the back door. If it still cannot go clean, the stop-and-surface rule fires exactly as before. Two stop-and-surface events have already occurred (passes 3 and 6); a third is the human's decision, not a reason to lower the bar. --- ## Global Constraints - **The hook always exits 0.** Every path, including a classifier failure, an unwritable marker and a failed emit. Withholding a *count* must never become a non-zero *exit*. -- **POSIX `sh` only.** No bash-isms. `shellcheck --shell=sh` is in the battery. POSIX `awk` is permitted (already POSIX; no new dependency). -- **`jq` is optional.** With and without it, every unambiguous payload must reach the same class. Where they cannot, the payload is `unrecognized` — never a guess. +- **Marker writes use `printf '%s' '' > f`, never `: > f`.** `:` is a POSIX **special builtin**: a redirection failure on one makes the shell exit, ignoring the enclosing `{ … } 2>/dev/null || true` and even an `if`. Verified 2026-08-01 — with a directory at the target, the `:` form prints nothing and exits **2** under `dash`, while the `printf` form exits 0. Ubuntu's `/bin/sh` **is** `dash`, and CI runs `ubuntu-24.04`. **This is a live defect in the shipped hook:** `codex-gate.sh:394` uses that form today, so on Linux an unwritable `.context/` makes the hook exit 2 from the unknown-tool branch — invariant 1 violated in released code. Task 4 fixes it. The existing "special-builtin redirection regression" test only ever ran under macOS `sh`, where the form survives, which is why it stood. +- **POSIX `sh` only.** No bash-isms. `shellcheck --shell=sh` is in the battery, and it applies to the suite as much as to the hook. +- **`awk` and `sed` are both required for classification; `jq` is irrelevant to it.** `awk` locates, `sed` runs the blank test. A failure in either is uncertainty, not a verdict: an `awk` status that is neither 0 nor 1, and a nonzero `sed`, both map to `unrecognized` — counted, disclosed, never a guess. `sed`'s status is checked for a specific reason: a failed substitution yields an empty string, which reads as blank and would classify a genuine success envelope as `no-result` — fail-**closed**, the one direction this design refuses for a result it can see. - **Loose in the firing direction.** On uncertainty, fire. A false ✓ is the dangerous direction. -- **Hook messages are prompts.** `docs/prompt-standards.md`, all 12 items, for every string added or changed. -- **Every plugin change requires a manifest version bump** (invariant 12) — Task 11. -- **Quality battery:** the `quality` row of `AGENTS.md` § Commands. Run it before every commit. -- **Never `git add -A`.** Stage the exact paths named in each task. +- **Hook messages are prompts.** `docs/prompt-standards.md`, all 12 items, for every string added or changed. The final strings are in Task 5 — Gate A cannot review a prompt that does not exist. +- **Never `git add -A`.** Stage the exact paths each task names, and **check the index first**: `git diff --cached --name-only` must contain nothing outside that list before committing. A pre-staged unrelated file otherwise enters the WIP commit and then the squash, silently. + +### The battery, per commit + +- **Before committing** — the full `quality` row of `AGENTS.md` § Commands **minus** `check-version-bump.sh`. That is six `shellcheck` invocations, **four** script runs and `claude plugin validate . --strict` — not three, as an earlier draft said; miscounting is how `check-invariants.test.sh` or `check-version-bump.test.sh` gets dropped while the task still claims the full battery. **This block, verbatim:** + +```sh +shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && \ +shellcheck --shell=sh --exclude=SC2015 plugins/dev-workflow/hooks/codex-gate.test.sh && \ +shellcheck --shell=sh scripts/check-invariants.sh && \ +shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && \ +shellcheck --shell=sh scripts/check-version-bump.sh && \ +shellcheck --shell=sh scripts/check-version-bump.test.sh && \ +sh plugins/dev-workflow/hooks/codex-gate.test.sh && \ +sh scripts/check-invariants.test.sh && \ +sh scripts/check-invariants.sh && \ +sh scripts/check-version-bump.test.sh && \ +claude plugin validate . --strict +``` + + Every later task means **this**, run and green — re-read it against the `AGENTS.md` row before the first commit, since that row is the source of truth and this is a copy. +- **Plus a `dash` run of the hook suite** — `dash plugins/dev-workflow/hooks/codex-gate.test.sh` — from Task 4 onward, where the marker writes begin. The special-builtin defect above is shell-dependent and macOS `sh` does not expose it; claiming `dash` coverage without running `dash` is what let it ship. If `dash` is unavailable, say so and record CI-on-Ubuntu as the only evidence, rather than claiming both. +- **After committing** — `sh scripts/check-version-bump.sh "$BASE"`, with `$BASE` the SHA recorded at Task 1. It compares commits, so the commit must exist first; and passing `main` on this checkout compares HEAD with itself and succeeds trivially, which is not evidence. ## File Structure | File | Responsibility | Task | |---|---|---| -| `plugins/dev-workflow/hooks/fixtures/*.json` | Sanitized captured payloads; the suite's only source of payload shapes | 1 | -| `plugins/dev-workflow/hooks/fixtures/README.md` | Provenance and the fact of sanitization | 1 | -| `plugins/dev-workflow/hooks/codex-gate.test.sh` | Helpers updated first, then coverage per task | 2, and every task after | -| `plugins/dev-workflow/hooks/codex-gate.sh` | `emit` status, locator, classifier, state effects, messages | 3–9 | -| `README.md`, `CLAUDE.md`, `commands/workflow-init.md` | The shipped statements this change falsifies | 10 | -| `plugins/dev-workflow/.claude-plugin/plugin.json`, `CHANGELOG.md` | Version bump and entry | 11 | +| `plugins/dev-workflow/.claude-plugin/plugin.json` | version bump `0.7.1` → `0.8.0` | 1 | +| `plugins/dev-workflow/hooks/fixtures/*.json` + `*.response.json` + `README.md` | sanitized captured payloads | 1 | +| `plugins/dev-workflow/hooks/codex-gate.test.sh` | the suite, written test-first against this plan's oracles | 2, and every task after | +| `plugins/dev-workflow/hooks/codex-gate.sh` | `emit` status, buffered output, locator, classifier, state effects, messages | 3–5 | +| `README.md`, `CLAUDE.md`, `commands/workflow-init.md`, `AGENTS.md`, `docs/architecture.md`, `codex-gate.sh` | the shipped statements this change falsifies, and the layout trees the new directory drifts | 6 | +| `plugins/dev-workflow/CHANGELOG.md` | the entry, written when the behaviour it describes exists | 7 | + +**Ordering rationale.** The manifest bump is in the **first** plugin-touching commit: `check-version-bump.sh` compares base against HEAD, so once any plugin file is committed without a bump every later task fails it. The CHANGELOG entry is not there — an intermediate commit carrying a released version whose entry describes absent behaviour is misleading history. Fixtures and helpers land before any behaviour change. `emit`'s status change and the output buffer land before the markers that depend on them. Documentation lands after the behaviour it describes is real. -**Ordering rationale:** fixtures and helpers land before any behaviour change, so the suite is green on both sides of the classifier. `emit`'s status change lands before the markers that depend on it. Documentation lands after the behaviour it describes is real. +**Every intermediate commit is `WIP:`-prefixed.** CLAUDE.md §5 is *tests green → Gate B → commit*, and the hook treats any non-`wip` commit as closing a Gate-B cycle: it resets the counters and fires a STOP on the next one. Six plain commits before the single Gate-B cycle would be six plugin changes that each closed a cycle without a review — and on a shared or interrupted branch they are what someone else pulls. Task 7 squashes them into one reviewed snapshot **before** Gate B runs. --- -### Task 1: Fixture home +### Task 0: Execution preconditions -**Files:** -- Create: `plugins/dev-workflow/hooks/fixtures/{shape0-success,shape1-fast-fail,shape2-executor-timeout,shape3-backgrounding-notice}.json` -- Create: `plugins/dev-workflow/hooks/fixtures/README.md` -- Source (untracked, do not ship): `.context/probe-payloads/` +Before any of this runs. Neither is optional, and both exist because the working tree at +Gate-A close is **not** clean. -**Interfaces:** -- Produces: `$FIXTURES` resolution rule used by every later task — `FIXTURES="$(dirname "$0")/fixtures"`. +- [ ] **Commit the approved Gate-A artifacts first, on their own.** The spec amendment (§3.1) + and this plan are unstaged edits right now. Left that way, they either fall outside every + WIP snapshot and the whole Gate-B range — the settled amendment invisible to the reviewer + — or get swept into the squash without appearing in the reviewed diff. Commit them as a + normal docs commit (`docs/**.md` is prose, so Gate B is N/A per CLAUDE.md §5), **then** + record `$BASE`. +- [ ] **Then stop on anything unaccounted.** `git status --porcelain` must be empty. A + stray edit or a pre-staged file otherwise enters the first WIP commit, then the squash, + and the pre-reset path audit in Task 7 cannot see what it never staged. +- [ ] **Record `$BASE = git rev-parse HEAD` here, after those commits and before Task 1**, + and write the literal SHA into a durable note — the execution log or the evidence entry + draft — not a shell variable. Tasks 1 and 7 both need it, in different shells, hours + apart; a variable that lived in one terminal is how Gate B ends up with the wrong range. + Validate it with `git merge-base --is-ancestor "$BASE" HEAD` before each use. -- [ ] **Step 1: Sanitize and copy the four existing captures** +### Task 1: Fixtures and the version bump -`tool_response` must stay byte-exact; only machine-specific metadata is neutralized. +**Files:** modify `plugins/dev-workflow/.claude-plugin/plugin.json`; create `plugins/dev-workflow/hooks/fixtures/` with seven payloads, their `*.response.json` slices, and `README.md`. Source (untracked, never shipped): `.context/probe-payloads/`. -```sh -cd /Users/daniel/DEVELOPMENT/APPS/dev-workflow-kit -mkdir -p plugins/dev-workflow/hooks/fixtures -for pair in \ - "shape0-success.json:shape0-success.json" \ - "shape1-fast-fail-execution-failed.json:shape1-fast-fail.json" \ - "shape2-executor-timeout.json:shape2-executor-timeout.json" \ - "shape3-backgrounding-notice.json:shape3-backgrounding-notice.json" -do - src=".context/probe-payloads/${pair%%:*}" - dst="plugins/dev-workflow/hooks/fixtures/${pair##*:}" - jq '.session_id="00000000-0000-0000-0000-000000000000" - | .transcript_path="/dev/null" - | .cwd="/tmp/fixture-repo" - | .tool_use_id="toolu_fixture" - | .prompt_id="00000000-0000-0000-0000-000000000000"' "$src" > "$dst" -done -``` +**Produces:** `$FIXTURES` — `"$(cd "$(dirname "$0")" && pwd)/fixtures"` — used by every later task. -- [ ] **Step 2: Verify `tool_response` survived byte-exact** +- [ ] **Step 1: Bump the manifest version.** `0.7.1` → **`0.8.0`** — minor, not patch: the hook changes what a gate call does to the counters, which is new behaviour under the same interface. Invariant 12's checker cannot judge that, so it is named here. **No CHANGELOG entry yet** — Task 7. -```sh -for pair in \ - "shape0-success.json:shape0-success.json" \ - "shape1-fast-fail-execution-failed.json:shape1-fast-fail.json" \ - "shape2-executor-timeout.json:shape2-executor-timeout.json" \ - "shape3-backgrounding-notice.json:shape3-backgrounding-notice.json" -do - a=$(jq -c '.tool_response' ".context/probe-payloads/${pair%%:*}") - b=$(jq -c '.tool_response' "plugins/dev-workflow/hooks/fixtures/${pair##*:}") - [ "$a" = "$b" ] && echo "OK ${pair##*:}" || { echo "DRIFT ${pair##*:}"; exit 1; } -done -``` +- [ ] **Step 2: Sanitize the four captures BY HAND.** Not with `jq`: rewriting a payload through `jq` reserializes `tool_response`, erasing exactly the escape and whitespace variants the matcher must read — while a `jq -c '.tool_response'` comparison would still report "identical", because it compares semantics, not bytes. -Expected: four `OK` lines. Any `DRIFT` means the sanitizer touched the object under test — stop and fix the filter. + **The seven fixtures, source to destination.** Verified against `.context/probe-payloads/` on 2026-08-02; note the one rename, which no earlier draft named and which every later inventory depends on: -- [ ] **Step 3: Capture the review-tool fixture against a disposable synthetic repo** + | Source | Destination | Origin | + |---|---|---| + | `shape0-success.json` | `shape0-success.json` | capture | + | `shape1-fast-fail-execution-failed.json` | **`shape1-fast-fail.json`** | capture, **renamed** | + | `shape2-executor-timeout.json` | `shape2-executor-timeout.json` | capture | + | `shape3-backgrounding-notice.json` | `shape3-backgrounding-notice.json` | capture | + | — | `shape0-success-review.json` | Step 4, synthetic-repo capture | + | copy of `shape0-success.json` | `collision-success-quotes-both.json` | Step 5, synthetic | + | copy of `shape1-fast-fail.json` | `collision-failure-quotes-true.json` | Step 5, synthetic | -`tool_response` cannot be redacted afterwards, so it must never contain real work. Create a throwaway repo with invented content, run one `mcp__codex__review` against it, and capture the payload with the same hook-instrumentation method recorded in `.context/probe-payloads/INDEX.md`. Save as `shape0-success-review.json`, sanitized by the Step 1 filter. + Each gets a sibling `.response.json` (Step 6), so fourteen files plus the README. Every later label, slice check, replay row and counterfactual uses **these** names. -**Then read the whole fixture end to end before staging it.** Confirm no real path, code excerpt or finding text appears. This is a manual gate; it has no automated check because the thing it guards against is content, not shape. + Edit **only** these values, changing no byte inside `tool_response`: -- [ ] **Step 4: Write the provenance README** + | Field | Replacement | Why | + |---|---|---| + | `session_id`, `prompt_id` | `00000000-0000-0000-0000-000000000000` | session identity | + | `transcript_path` | `/dev/null` | absolute path under `~` | + | `cwd`, `tool_input.workingDirectory` | `/tmp/fixture-repo` | machine layout; the second carries a private scratchpad path | + | `tool_use_id` | `toolu_fixture` | session identity | + | `tool_input.instruction` | `probe: reply with ok` | prompt content across a trust boundary | -```markdown -# Hook test fixtures + `permission_mode` and `effort` carry no machine or prompt data and stay as captured. The `tool_input` **object is kept**, not emptied: this repo's own gate prompts quote payload text, so a realistic `tool_input` is what makes the decoy tests mean anything. -Real `PostToolUse` payloads captured from the hook, used by `codex-gate.test.sh`. +- [ ] **Step 3: Prove `tool_response` survived byte-exact.** Extract the located block from source and copy with the same locator, and compare. **Requirements on the procedure**, since they are what earlier drafts got wrong: both extractions must be checked for **status 0** before their outputs are compared (two failed extractions produce two equal empty strings and would print `OK`); the step must **exit nonzero** on any mismatch (a bare `echo STOP` exits 0); and `diff`'s status must be accepted at 0 or 1, with only a real error failing it. Run Steps 3 and 7 in **one shell**, with the locator materialized once and cleaned up once — assuming a temp file survives between checklist steps in different shells is how this breaks. -**Sanitized, and how:** `session_id`, `transcript_path`, `cwd`, `tool_use_id` and -`prompt_id` are replaced with fixed placeholders. **`tool_response` is byte-exact** — -it is the object under test, so it is never rewritten. + The locator body is **not** in this plan (see the scoping note). Materialize it from `.context/plan-drafts/locate.awk`, or reconstruct and re-verify it against `verify.sh` first. -`shape0-success-review.json` was captured against a disposable synthetic repository -with invented content, because a real review response embeds file paths, code and -findings that byte-exactness would preserve. + **Then read every `diff` yourself.** Only the Step-2 fields may differ. The automated half proves the located block is byte-identical; the human half catches a field nobody listed. The README states the bound: what is pinned byte-exact is the **located text block**, not every byte of the surrounding array. -| File | Class it exercises | -|---|---| -| `shape0-success.json` | `success` (exec) | -| `shape0-success-review.json` | `success` (review) | -| `shape1-fast-fail.json` | `failure` — `CODEX_EXECUTION_FAILED` | -| `shape2-executor-timeout.json` | `failure` — `CODEX_TIMEOUT` | -| `shape3-backgrounding-notice.json` | `backgrounded` | -``` +- [ ] **Step 4: Capture the review-tool fixture — in an isolated install.** `tool_response` cannot be redacted afterwards, so it must never contain real work: capture against a **disposable synthetic repository** with invented content. -- [ ] **Step 5: Run the battery and commit** + **Isolation is mandatory, not preferred.** The method in `.context/probe-payloads/INDEX.md` inserts a stdin dump into the *globally installed* cached hook, which serves every project on the machine and would record complete payloads — prompts, absolute paths, review content — from any concurrent work. Use a separate Claude Code profile or a scratch plugin install. **If isolation cannot be established, stop and surface** rather than mutating the shared cache: the fixture is worth less than the exposure. -```sh -sh plugins/dev-workflow/hooks/codex-gate.test.sh && sh scripts/check-invariants.sh -git add plugins/dev-workflow/hooks/fixtures -git commit -m "test(hooks): ship sanitized captured payloads as fixtures" -``` + Should a human later decide the shared-cache route is acceptable anyway, the procedure needs all of: a guarded backup whose checksum status is checked directly (`cmd | cut` reports `cut`'s status, so a partial checksum reads as verified); **one** cleanup function, idempotent, that restores and verifies before releasing the backup; signal handlers that **exit** after cleanup; no `trap -` before the restore; and deletion of every payload the dump captured for a call other than this one. ---- + **Then read the whole fixture end to end before staging it.** No real path, code excerpt or finding text. A manual gate — the thing it guards is content, not shape. Same read for the four sanitized captures. -### Task 2: Existing helpers carry real results +- [ ] **Step 5: Create the two collision fixtures.** Copy `shape0-success.json` and `shape1-fast-fail.json`; edit the `summary` inside the result text so each quotes **both** marker literals. **Synthetic by necessity** — no real call produces them — and the README says so. They pin the failure direction: a success quoting `false` stays `success`, a failure quoting `true` stays `failure`. -Under this design a payload with no `tool_response` is `no-result` and stops counting. Every existing counting call site sends exactly that. They are updated **before** the classifier lands, so the suite is green on both sides. +- [ ] **Step 6: Write the response slices.** For each fixture, a sibling `.response.json` holding only that fixture's `tool_response` array, byte-identical to the slice inside the payload. Copy by hand; do not re-serialize. -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.test.sh:33-38` (helpers), plus the direct payloads at lines ~225, ~365, ~369 and the mapped-tool payloads at ~465, ~481 + This is the one representation choice, made here rather than left open: the driver must build payloads **without `jq`**, or a machine with no `jq` fails the driver rather than the hook — and `sed`-extracting a multi-line array at test time is a second parser nobody reviews. The cost is one duplicated slice per fixture, which Step 7 pins. -**Interfaces:** -- Produces: `rev()`, `execp()`, `codextool()` all carrying a success envelope; `rev_noresult()` for the deliberately result-less case. +- [ ] **Step 7: Prove each slice matches its payload**, in the same single shell as Step 3, exiting nonzero on any drift. What it compares is the **located block**, not the whole array: two slices differing only in inter-element whitespace would pass, and the README must not claim more. Its **permanent** form belongs in Task 5, where the locator lives in the hook and the suite can drive it with no untracked reference — a permanent test reading an ignored path is green here and broken in every clone. -- [ ] **Step 1: Add the fixture path and envelope helpers** +- [ ] **Step 8: Write the provenance README.** What was sanitized and how; that `tool_response` is byte-exact and *what that claim covers*; why the review fixture came from a synthetic repo; which fixtures are synthetic; why the slices exist and what pins them; one row per fixture naming the class it exercises. -```sh -FIXTURES="$(dirname "$0")/fixtures" -# The success envelope every "a gate call happened" helper now carries. -succ() { jq -c '.tool_response' "$FIXTURES/shape0-success.json"; } -payload() { # $1 = tool name, $2 = tool_response JSON - printf '{"hook_event_name":"PostToolUse","tool_name":"%s","tool_input":{},"tool_response":%s}' "$1" "$2" -} -``` +- [ ] **Step 9: Battery and commit.** Write out the full pre-commit block here (see § The battery), then check the index, stage exactly `plugins/dev-workflow/.claude-plugin/plugin.json` and `plugins/dev-workflow/hooks/fixtures`, commit as `WIP: test(hooks): ship sanitized captured payloads as fixtures (0.8.0)`, and run `sh scripts/check-version-bump.sh "$BASE"` afterwards. -- [ ] **Step 2: Rewrite the helpers to use it** + `$BASE` was recorded in Task 0, **before** this commit — not derived here as `HEAD~1`, which is wrong the moment a fix adds a commit, and not used before it exists, which is what an earlier draft did by calling the version checker with an unset variable. -```sh -rev() { run "$(payload mcp__codex__review "$(succ)")" >/dev/null; } -execp() { run "$(payload mcp__codex__exec "$(succ)")" >/dev/null; } -codextool() { run "$(payload "$1" "$(succ)")"; } -rev_noresult() { run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}'; } -``` +--- -`rev_noresult` exists so the `no-result` tests state their case rather than inheriting it. +### Task 2: The suite harness -- [ ] **Step 3: Replace every direct result-less gate payload** +Every helper later tasks rely on, defined before first use so no section aborts under `set -u`. **The shell is written here, test-first, not copied from this plan** — `.context/plan-drafts/harness.sh` is an unrun seed whose header lists the three problems it is known to carry. -Search and convert: +**What the harness must provide**, by contract: -```sh -grep -n '"tool_name":"mcp__codex__\(exec\|review\)"' plugins/dev-workflow/hooks/codex-gate.test.sh -``` +| Helper | Contract | +|---|---| +| `payload`, `resp`, `resp_from`, `resp_success`, `unrec` | build payloads and responses with **`printf` and `cat` only** — no `jq`, or the driver becomes the thing under test | +| `payload_from` | retarget a whole captured fixture to another tool name, touching one field and no `tool_response` byte | +| `run`, `rev`, `revout`, `execp`, `codextool`, `codextool_unrec`, `rev_noresult` | capturing and silent runners, **named for which they are** — a silent runner behind a message assertion makes it vacuous | +| `run_closed`, `nojq_run`, `nojq_run_closed` | stdout closed, and the jq-free `PATH`, in both combinations | +| `mk_path` | restricted `PATH` builders. Must pass `shellcheck --shell=sh`: the obvious `mk_path nojq $HOOK_CMDS` form splits an unquoted expansion (SC2086) and a command substitution (SC2046) | +| `reset_all`, `reset_gate_state` | full reset **including the opt-out marker**, and a narrow gate-state-only reset for scenario sequencing | +| `class_of` | one invocation's class from **that invocation's own** effects — counters, markers, and an exact message substring. No cross-test global state | +| `run_scenario` | one hook invocation per branch name, with per-scenario setup and cleanup | +| `field_of`, `golden` | exact field comparison, **consistently** handling the jq-free case in every caller | -Every hit that represents *a gate call that should count* becomes `execp`/`rev`/`payload …`. The only hits left without `tool_response` are inside `rev_noresult`. +**Two contracts the restricted `PATH`s must satisfy, because getting them wrong makes whole sections pass vacuously:** -- [ ] **Step 4: Run the suite against the UNCHANGED hook** +1. **Every external command the hook runs must be linked**, not just the one being tested. `tree_hash` shells out to `mktemp` and `cp`; without them every jq-free Gate-B scenario computes `unavailable` and takes a different branch, so a matrix claiming to compare `jq` and jq-free would compare two different code paths. **Oracle:** under the jq-free `PATH`, a success fixture records a **usable, self-matching** fingerprint. If it does not, the jq-free rows prove nothing and must not be reported as coverage. +2. **A fault shim must fail only what it targets.** Removing `sed` entirely breaks `field()`'s routing, so the hook never reaches `emit` and an encoder-failure test passes for an unrelated reason. Every shim is verified in both directions — the normal path still works, the targeted path fails — before any assertion depends on it, and a shim that cannot be built prints `skip -` **with its dependent assertions skipped too**, not left to run. -Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh` -Expected: `all passed`. The hook ignores `tool_response` today, so adding it changes nothing — that is the point: this task is behaviour-neutral and provable. +**Then convert the existing suite:** every payload representing *a gate call that should count* carries a real success envelope, including the mapped-tool sections. The only result-less payloads left are inside `rev_noresult`. -- [ ] **Step 5: Commit** +**Oracle for this whole task:** the suite passes against the **unchanged** hook. The hook ignores `tool_response` today, so adding it changes nothing — which is what makes this task behaviour-neutral and provable. -```sh -git add plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "test(hooks): drive gate calls with real result envelopes" -``` +Commit: `WIP: test(hooks): drive gate calls with real result envelopes`. --- ### Task 3: `emit` propagates writer status -Spec §6 defines marker-writing as conditional on "a complete hook JSON document was written". `emit` currently returns 0 unconditionally after its output command. +Spec §6 defines marker-writing as conditional on "a complete hook JSON document was written". `emit` returns 0 unconditionally after its output command. -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — `emit()` -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +**Produces:** `emit` returns `0` written · `1` suppressed by the off-switch · `2` write failed. Callers treat **only 0** as "shown". -**Interfaces:** -- Produces: `emit` returns `0` written, `1` suppressed by the off-switch, `2` write failed. Callers treat **only 0** as "shown". +- [ ] **Step 1: Implement.** Both `jq` and fallback branches return 2 when their writer fails. In the fallback, **both `sed` encoder substitutions must succeed or return 2**: a failed substitution yields an empty field while `printf` still exits 0, so without the check a truncated document reports "written" and burns a one-shot on a message nobody can read. -- [ ] **Step 1: Write the failing test** +- [ ] **Step 2: B2's hook-side occurrence, in the same edit.** `codex-gate.sh:291` says state tracking "keeps running so re-enabling is accurate". Replace with: *"State tracking keeps running while off, so re-enabling carries the same counting semantics as if the gate had been on — not a guarantee that every counted call was reviewed."* -```sh -# Section: emit reports writer failure -reset_all -: > "$off_file_unused" 2>/dev/null || true -out=$(rev 2>/dev/null >&-; echo "rc=$?") # stdout closed -printf '%s' "$out" | grep -q 'rc=0' && pass "hook still exits 0 with stdout closed" \ - || fail "hook still exits 0 with stdout closed" -``` +**Oracles:** -- [ ] **Step 2: Run it to see it fail** +| Label | Must fail on | +|---|---| +| `hook exits 0 with stdout closed` | any path where a dead stdout propagates a nonzero exit | +| `a failed write does not burn the one-shot` | `emit` returning 0 after a failed write. Asserted through the **unknown-tool note**: a successful review `PostToolUse` emits nothing, so closing stdout on it exercises no writer at all and both outcomes hold before the change | +| `jq-free: failed write does not burn the one-shot` | the same, through the fallback emitter | +| `encoder failure exits 0, prints nothing, burns nothing` | an empty-but-well-formed document being written and treated as success. Needs the **selective** `sed` shim | -Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -i 'stdout closed'` -Expected: FAIL — nothing distinguishes a failed write yet. +Commit: `WIP: fix(hooks): emit reports whether it actually wrote`. -- [ ] **Step 3: Make `emit` report its writer** +--- -```sh - [ -f "$off_file" ] && return 1 - if command -v jq >/dev/null 2>&1; then - jq -cn --arg ev "$event" --arg ctx "$1" --arg msg "$2" \ - '{hookSpecificOutput:{hookEventName:$ev,additionalContext:$ctx},systemMessage:$msg}' || return 2 - else - ctx=$(printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g') - msg=$(printf '%s' "$2" | sed 's/\\/\\\\/g; s/"/\\"/g') - printf '{"hookSpecificOutput":{"hookEventName":"%s","additionalContext":"%s"},"systemMessage":"%s"}\n' \ - "$event" "$ctx" "$msg" || return 2 - fi - return 0 -``` +### Task 4: One emit per invocation — discharges A7, and A6's mechanism -- [ ] **Step 4: Fix the stale comment in the same edit** +Spec §6: an invocation owing two messages composes them into one document. Impossible while each branch writes as it decides, so output is **buffered** and written once at the end. **Behaviour-neutral**: the same branches say the same things through a different pipe. -The `emit` header says state tracking "keeps running so re-enabling is accurate". Spec §5.2 retires that word. Replace with: *"State tracking keeps running while off, so re-enabling carries the same counting semantics as if the gate had been on — not a guarantee that every counted call was reviewed."* This is item **B2**'s hook-side occurrence; the other two sites are Task 10. +**Produces:** `note ctx msg` (append) and `flush_notes` (write once, then apply one-shot markers). `emit` is called from `flush_notes` and nowhere else. -- [ ] **Step 5: Verify and commit** +**A7 — separator and encoding.** `additionalContext` bodies join with `" — "` (space, em dash, space); `systemMessage` bodies with a single space. **No newline anywhere** — the `jq`-free emitter escapes only backslash and quote, and a literal newline would produce an invalid JSON document. A disclosure carried from an earlier event is prefixed `Earlier: ` in **both** fields, so the model-facing and user-facing copies cannot disagree about which call the statement is about. -Run the battery. Expected: `all passed`, `shellcheck` clean. +- [ ] **Step 1: Add the buffer and convert all nine call sites.** Every `emit "…" "…"` becomes `note "…" "…"` with identical strings. The one site that read `emit`'s status — the unknown-tool note — sets a `mark_noted` flag that `flush_notes` applies only on status 0. `flush_notes` runs immediately before the final `exit 0`; both earlier `exit 0`s (not adopted, not a git repo) are before any `note`, so nothing is buffered when they fire. -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "fix(hooks): emit reports whether it actually wrote" -``` +- [ ] **Step 2: Convert `codex-gate.sh:394` to the `printf` form** as part of moving it — this is the live invariant-1 defect in § Global Constraints. ---- +**Oracles:** -### Task 4: Locate the result block — discharges A1, A2 +| Label | Must fail on | +|---|---| +| ` emits exactly one document`, for all nine emitting branches | zero output as well as two. **`= 1`, never `-le 1`** — `-le 1` passes a dropped message, which is the failure this assertion exists to catch | +| `a silent event emits nothing` | any output on a reachable non-emitting event | +| the existing suite, unchanged | any behavioural difference from the conversion, including *"suppressed note does not burn its one-time marker"* | +| `exits 0 with a directory at the marker path`, **under `dash`** | the special-builtin exit. macOS `sh` does not expose it | -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — new `locate_result()` above the `case "$event"` dispatch -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +**`silent` must be an event the hook actually receives.** `hooks.json` registers **two different matchers** — verified 2026-08-02: `PostToolUse` is `^(Bash|Skill|mcp__codex__.*)$` and `PreToolUse` is the unanchored `Bash|Skill`. It is the **PostToolUse** one that matters here, and under it a `PostToolUse` for `Edit` never arrives in production, so a debt-flush test built on it proves nothing about a reachable invocation. Use a **non-commit Bash `PostToolUse`**. Cite the event when citing the matcher: the two are not the same string, and `run_scenario` drives both events. -**Interfaces:** -- Produces: `locate_result` prints the located block's text **in escaped form** on stdout and returns `0`; returns `1` for *unambiguously nothing there* (→ `no-result`); returns `2` for *cannot determine* (→ `unrecognized`). Consumed by Task 5. +Commit: `WIP: refactor(hooks): buffer output so one invocation writes one document`. -**A1 — the scanner state machine, stated operationally.** The scanner walks the payload one character at a time holding three variables: `instr` (inside a JSON string), `esc` (the previous character was an unescaped backslash), and `depth` (object/brace nesting, counted only while `instr` is 0). **Backslash parity** falls out of `esc` toggling rather than counting: each backslash flips it, any other character clears it, and a quote closes the string only when `esc` is 0. **"Depth 1"** means: the key's opening quote occurs while `depth == 1`, i.e. directly inside the payload's single top-level object. +--- -**A2 — recognition.** A second depth-1 `tool_response` key is *ambiguity*, returning 2. Malformed JSON is **not** classified: if the outer document cannot be walked to a balanced end, the scanner returns 2 and — per spec §3.3 — the hook has already failed to route the event at all, so nothing is emitted. +### Task 5: Classify and act — discharges A1–A5, completes A6 -- [ ] **Step 1: Write the failing tests, one per contract case** +**One task and one commit, deliberately.** Splitting locator, classifier, wiring and disclosure produced intermediate commits that could not be green: the classifier is unobservable until it is wired, and the wiring calls the diagnostic interface. Each such commit would ship either a function nothing calls or a call to a missing function. -```sh -# Section: locate_result -lr() { printf '%s' "$1" | sh "$HOOK_LOCATE"; } # thin harness exposing locate_result +**A1 — the state machine, stated operationally.** The scan holds the whole payload in one string and walks it with three primitives. `readstr` consumes a JSON string from its opening quote and returns the index of its closing quote, accumulating the **raw** bytes: **backslash parity falls out of an `esc` flag** — a backslash takes the next byte verbatim whatever it is, so `\\` ends parity and the following `"` closes the string, while `\"` does not. `skipws` consumes space, tab, newline and CR. `skipval` consumes one **span**: a quote-aware string; a balanced container span whose closers must match their openers (`[1}` is rejected); or a primitive, only if the token is exactly `true`, `false`, `null` or a JSON number. It does **not** parse a container's members. **"Depth 1" is not a counter** — it is the structural position the walk occupies: the top-level loop reads key/colon/value triples of the payload's single outer object and nothing else, so a `tool_response` inside any string or nested container is never a candidate. -t='{"tool_input":{"instruction":"see \"tool_response\" docs"},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' -lr "$t" | grep -q '\\"success\\": true' && pass "ignores tool_response quoted in tool_input" \ - || fail "ignores tool_response quoted in tool_input" +**A2 — recognition, and what carries the safety. `locate_result` is a locator, not a validator**, and this paragraph claims nothing more. Three passes read a stronger promise into the prose than the code keeps; the promise is stated at its true size and pinned by fixtures rather than by wording. -t='{"tool_response":[{"type":"image","data":"x"},{"type":"text","text":"{\"success\": false}"}]}' -lr "$t" | grep -q '\\"success\\": false' && pass "skips a non-text block" || fail "skips a non-text block" +**What makes the walk trustworthy is string-boundary tracking, and nothing else.** `readstr` decides where every JSON string starts and ends, from quote state and backslash parity alone. That one property is why a `tool_response` mentioned *inside* a string — the `tool_input` decoy, a result quoting the key, this repo's own gate prompts — is never mistaken for the key. **Malformation outside a string boundary cannot redirect the walk**: it may be stepped over, but it cannot move where the next string begins. -t='{"tool_response":[{"type":"text","text":"a"}],"tool_response":[{"type":"text","text":"b"}]}' -lr "$t"; [ $? -eq 2 ] && pass "duplicate depth-1 key is ambiguous" || fail "duplicate depth-1 key is ambiguous" +**Refusals it makes**, each tested: a second depth-1 `tool_response` key; a repeated `type` or `text` member in **any element the scan examines**, not only the one it would select (verified: a duplicate inside a non-text element *preceding* a valid text block also refuses — stricter than the first-text-element rule requires, and it is the behaviour, so the prose states it); a document not starting with `{`; a non-string key, missing colon or unterminated string in the path it walks; a mismatched container delimiter; a stray comma; a bare token that is not a JSON literal; garbage after the outer object's `}`; and **nesting deeper than 200**. -t='{"tool_response":[]}' -lr "$t"; [ $? -eq 1 ] && pass "empty array is nothing-there" || fail "empty array is nothing-there" +**Refusals it does NOT make, frozen as fixtures asserting today's behaviour.** A *walkable* invalid document is walked past and the real block located, status 0 — `[1,]`, `{"a" 1}` and `"\q"` in a sibling value are pinned so this correspondence is checked mechanically instead of argued again. The response array is walked only as far as the selected element. An escaped key spelling is compared as raw bytes and simply does not match. -t='{"tool_response":{"type":"text","text":"x"}}' -lr "$t"; [ $? -eq 1 ] && pass "non-array container is nothing-there" || fail "non-array container is nothing-there" -``` +**Why that is the right size.** The payload's producer is Claude Code, whose serializer emits valid JSON, so a walkable-invalid document is synthetic. And the threat a validator would answer is one **spec §10 already accepts by name**: a mapped third-party tool is trusted for counting, so a hostile server needs no malformed JSON — it can return `{"success": true}`. -- [ ] **Step 2: Run them to see them fail** +**Two bounds, and what each does and does not cover.** The **length ceiling** refuses a payload whose accumulated length would exceed 1 Mi *`awk` length units* — not bytes: POSIX `awk`'s `length()` counts characters and implementations differ on multibyte input, so the cut-off is not identical between BWK `awk` and `mawk`. It bounds the **scan**, not memory: `payload=$(cat)` already holds the whole input and `awk` still ingests it. The **depth cap** refuses nesting past 200, because the closer stack is a string and each push is O(length), making deep nesting quadratic in work an external MCP result could dictate. Neither is a contract; both are backstops, and both route to `unrecognized`. -Run: `sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -c '^FAIL'` -Expected: 5 failures — `locate_result` does not exist. +Malformed JSON is **not a class**: per spec §3.3 a payload the hook cannot *route* never reaches here. **Routability itself depends on `jq`, and that predates this change**: with `jq`, `field()` on a malformed document returns empty and the hook exits silently; without it, the `grep` fallback can still read a tool name, so the same payload routes and lands in `unrecognized`. That divergence belongs to `field()` and is left alone — stated because a reader comparing the two environments will otherwise read it as a classifier bug. -- [ ] **Step 3: Implement the scanner** +**A3 — the accepted encodings, complete.** Whitespace is accepted at exactly **three** points — after the encoded `{`, after the key `\"success\"`, after the `:` — and at each: literal ASCII space, and the two-byte escapes `\n`, `\t`, `\r`. Nothing else; a Unicode escape is not whitespace here and needs no special handling, because it matches no anchor and the terminal default carries it to `unrecognized`. **Blank** is the same alphabet over the whole block. **The value token must end** at a delimiter — `true` followed by anything other than encoded whitespace, `,` or `}` is `unrecognized`, because `true*` alone accepts `truely`, a false **success**. Whitespace stripping is bounded at 64 units; past it the prefix match fails and the block is `unrecognized`. The blank test is one `sed` pass, so a large blank block costs linear work in `sed` rather than shell iterations. -```sh -# Locates the depth-1 "tool_response" value and returns the first text block's -# text in ESCAPED form. Exit 0 = located; 1 = unambiguously nothing there; -# 2 = cannot determine (ambiguous / malformed). -locate_scan() { - printf '%s' "$payload" | awk ' - BEGIN { RS = "\0"; found = 0; dup = 0 } - { - s = $0; n = length(s); depth = 0; instr = 0; esc = 0; keyq = 0 - for (i = 1; i <= n; i++) { - c = substr(s, i, 1) - if (instr) { - if (esc) { esc = 0 } - else if (c == "\\") { esc = 1 } - else if (c == "\"") { instr = 0; if (keyq) { key = buf; keyq = 0 } } - else if (keyq) { buf = buf c } - continue - } - if (c == "\"") { instr = 1; if (depth == 1) { keyq = 1; buf = "" } ; continue } - if (c == "{" || c == "[") { depth++; continue } - if (c == "}" || c == "]") { depth--; continue } - if (c == ":" && depth == 1 && key == "tool_response") { - if (found) { dup = 1; break } - found = 1; vstart = i + 1 - key = "" - } - } - if (dup) { exit 2 } - if (!found) { exit 1 } - print substr(s, vstart) - exit 0 - }' -} -``` - -The printed remainder is handed to a small block-picker that walks the array for the first `"type":"text"` element and prints its raw `"text"` value — still escaped, because nothing decoded it. +**A4 — the notice grammar, and what is deliberately outside it.** The anchor is exactly: the block **begins** `MCP tool \"` (escaped quote — the shared representation carries `\"`, and a draft expecting a literal `"` did not match the real capture at all), and the segment `\" is still running after ` occurs **before any `\n` escape**. Everything else is variable **by decision**: spec §4 places the quoted tool name, the threshold digits and unit, and the task id outside the anchor so it covers both gate tools, any mapped name and any threshold. Validating a duration format would narrow the anchor to shapes observed once, widening C1 rather than closing it. **Near-misses that must not match**, each tested: the phrase later in the text; a whole notice quoted inside a real envelope's `summary`; a block beginning `MCP tool \"` with no `is still running after`; the segment appearing only after a `\n`. -- [ ] **Step 4: Implement the `jq` path with its span check** +**A5 — the complete marker table.** Three files: `codex-gate.bgAdvice`, `codex-gate.unverified` (shown), `codex-gate.unverifiedPending` (owed). `bgAdvice` is independent of the other two. -```sh -locate_jq() { - blk=$(printf '%s' "$payload" | jq -r ' - if (.tool_response | type) != "array" then empty - else ( .tool_response[] | select((type == "object") and (.type == "text") and ((.text | type) == "string")) | .text ) - end' 2>/dev/null | head -n1) || return 2 - [ -n "$blk" ] || return 1 - enc=$(printf '%s' "$blk" | jq -Rs . | sed 's/^"//; s/"$//') - # Spec §3.1: exactly one occurrence, and inside the located span. - occ=$(printf '%s' "$payload" | grep -o -F "$enc" | wc -l | tr -d ' ') - [ "$occ" = 1 ] || return 2 - printf '%s' "$enc" -} -``` +| `unverified` | `pending` | Event | Result | +|---|---|---|---| +| absent | absent | `unrecognized`, flush wrote (0) | write `unverified` | +| absent | absent | `unrecognized`, flush suppressed (1) | write `pending` | +| absent | absent | `unrecognized`, flush failed (2) | write `pending` | +| absent | absent | `unrecognized`, flush wrote, **`unverified` write fails** | write `pending` — the debt survives the marker | +| absent | present | any unsuppressed event | prepend `Earlier: ` disclosure; on 0 → write `unverified`, delete `pending`; on 1 or 2 → **retain** `pending` | +| absent | present | the current event is **itself `unrecognized`** | one disclosure, **no `Earlier:` prefix**, pending cleared on a successful write | +| absent | present | flush wrote, `unverified` write fails | **retain** `pending` (duplicate beats loss) | +| absent | present | flush wrote, `unverified` written, **`pending` delete fails** | present+present → next row | +| present | present | any event | delete `pending` best-effort at flush start; no disclosure | +| present | absent | `unrecognized` again | nothing emitted, nothing written | +| absent | absent | `unrecognized`, flush **suppressed** and `pending` write fails | counted, nothing delivered, nothing recorded — **C2, direction 1** | +| absent | absent | `unrecognized`, flush **failed** and `pending` write fails | counted, nothing delivered, nothing recorded — **C2, direction 2** | +| any | any | any other marker write fails | proceed, exit 0; the debt is retained or retried per the rows above | +| `bgAdvice` absent | — | `backgrounded`, flush wrote | write `bgAdvice` | +| `bgAdvice` absent | — | `backgrounded`, flush suppressed or failed | **not** written — the one-shot stays unspent | +| `bgAdvice` present | — | `backgrounded` | short form; no write | -- [ ] **Step 5: Run the tests to verify they pass, in BOTH parser environments** +**The prefix is about whose call the statement describes**, not where the debt came from. `Earlier: ` exists so a carried disclosure cannot read as a statement about the current call; when the current call is *itself* `unrecognized`, the statement **is** about it, so no prefix — one message covers both and pending clears. That is why `note_unverified` setting the flag takes precedence over the pending check. -```sh -sh plugins/dev-workflow/hooks/codex-gate.test.sh -PATH=/usr/bin:/bin sh -c 'command -v jq >/dev/null && echo "jq present"; sh plugins/dev-workflow/hooks/codex-gate.test.sh' -``` +**C2 is two rows, not a property of marker writes generally.** A failed `bgAdvice` write costs a repeated advice message; a failed shown-write followed by a successful pending write keeps the debt and repeats the disclosure; a failed pending *delete* leaves the coexistence row, cleaned up next flush. None is a silent counted pass. Only **undelivered and unpersisted** produces one — in either of spec §5.2's two directions. -Expected: `all passed` in both. Follow the suite's existing `jq`-absent pattern (a stub `jq` earlier on `PATH` that fails) for the second run. +Concurrency is **not** covered by this table and is not meant to be: that is **C4**. -- [ ] **Step 6: Extraction parity assertion** +- [ ] **Step 1: Declare the message strings.** Final text, in the hook, above the branches. **Five pairs, ten strings.** All declared unconditionally so `set -u` cannot abort on any path. -For every fixture, both locators must hand the matcher **byte-identical** input. Assert the input, not the final class — two locating bugs can cancel out in the class. + **The field split decides where every sentence goes.** Spec §6: `additionalContext` is read by **Claude via Claude Code**; `systemMessage` by **the operator**. A remedy only a human can perform — restarting Claude Code, editing a config, changing a server timeout, unmapping a tool — belongs in `systemMessage`, because the model receiving `additionalContext` cannot do any of it. -```sh -for f in "$FIXTURES"/*.json; do - a=$(payload_from "$f" | with_jq locate_result) - b=$(payload_from "$f" | without_jq locate_result) - [ "$a" = "$b" ] && pass "extraction parity: $(basename "$f")" || fail "extraction parity: $(basename "$f")" -done -``` + **Structure (item 5):** each `additionalContext` carries compact tagged sections — ``, ``, ``, `` — rather than a flowing paragraph with inline labels, which is what item 5 asks for and what earlier drafts only approximated. Tags are literal text inside a single-line shell string; **no literal newlines**, per A7. Each `systemMessage` reads *state → operator action* **where there is one** — `FAILURE_MSG` carries state plus the one configuration remedy an operator owns, and nothing more. -- [ ] **Step 7: Commit** + **Item 1** is satisfied inside the delivered text: every `additionalContext` opens `Claude Code gate hook —`. A source comment naming the consumer is not the prompt. The header comment stays for the "checked that model's prompting page" half, which a prefix cannot carry. ```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "feat(hooks): locate the result block in both parser environments" -``` +FAILURE_CTX='Claude Code gate hook — this Codex call returned an envelope reporting failure. Not counted as a gate pass, no review fingerprint stored, does not count toward the floor. Read error.code in the tool result. CODEX_EXECUTION_FAILED is the pinned server generic failure code and does NOT tell you whether the call started, so check the accompanying error message and any session artifacts before assuming nothing ran; a call that did start may have left work behind. CODEX_TIMEOUT means the executor gave up mid-run: re-run the SAME call with the SAME scope. Any other code, or no code at all, is unclassified: every envelope whose first property is success false reaches this state, not only the two codes named here, so re-run once with the same scope and, if it repeats, report the code and message verbatim together with the effective server name and version from claude mcp list — an unfamiliar code is itself evidence about which server answered. Never retry with a narrower instruction or a smaller range, because that would count a pass for less than the artifact or diff the gate requires. Report one line: "gate pass discarded | error-code | started yes/no/unknown". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the retry also fails, stop and surface that line; the operator note carries the configuration remedy.' +FAILURE_MSG='⚠ Codex call failed — not counted as a gate pass. If the code was CODEX_TIMEOUT, the fix is configuration and only you can apply it: raise the executor timeout for the Codex MCP server, or reduce load outside the review. Do not ask for a smaller review scope — a narrower pass is worth less than a slow one.' ---- +NORESULT_CTX='Claude Code gate hook — this gate call carried no result text the hook could read. Not counted as a gate pass, no review fingerprint stored. Treat the pass as not run and report it. Two causes produce this shape and the tool name cannot separate them: a hooks-API payload change, or a mapped third-party tool returning empty or non-text content. Report one line: "gate call unreadable | mapped yes/no from .context/codex-gate.tools | claude-code version". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. One retry per pass is the whole recovery budget under CLAUDE.md section 5. A repeat is configuration or contract, not a transient failure, so stop and surface it; the operator note carries both checks.' +NORESULT_MSG='⚠ Gate call returned no readable result — not counted. Run both checks before concluding. First: does .context/codex-gate.tools map a tool name? Second: what does `claude mcp list` show as the effective server and version — not what .mcp.json says, because scope precedence can make a different entry of the same name effective. These checks narrow the cause; they do not prove it. If a mapping or a third-party server is in play, that tool may be returning empty or non-text content, which it can do legitimately: unmap it, or replace it with a server exposing exec and review. If both checks show the pinned server at its pinned version, a payload-contract change is the remaining explanation — record your Claude Code version and report it.' -### Task 5: Classify — discharges A3 +BG_LONG_CTX='Claude Code gate hook — this gate call was moved to the background at the auto-background threshold, 120 s by default, so its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. The original call may still be running and can still write its findings file later. Stop it by the task id in the tool result, or wait for it to finish, before deleting that slot or re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. Do not re-run while that task is active: a late writer landing in a slot you already re-ran leaves a correctly terminated file from the wrong run, and no downstream check can detect that. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it as a setup problem rather than retrying again.' +BG_LONG_MSG='⚠ Gate pass discarded (backgrounded) — a setup gap, not a failed review. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from, then restart Claude Code: it reads the value at process start, so exporting it inside a tool shell leaves the running session unchanged. Use 0 to disable auto-backgrounding, or a positive value that exceeds your longest gate call, since a positive value shorter than the call still backgrounds it. Requires Claude Code 2.1.212 or newer.' -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — new `classify()` -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +BG_SHORT_CTX='Claude Code gate hook — this gate call was backgrounded and its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. Stop or await the original call by the task id in the tool result before re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. Do not re-run while that task is active, so a late writer cannot land in a slot you already re-ran. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it rather than retrying again.' +BG_SHORT_MSG='⚠ Gate pass discarded (backgrounded) — not counted. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code; the full guidance was shown once earlier in this workspace.' -**Interfaces:** -- Consumes: `locate_result` (Task 4). -- Produces: `classify` prints one of `success|failure|backgrounded|no-result|unrecognized`. Consumed by Task 6. +UNVERIFIED_CTX='Claude Code gate hook — this workspace has classified at least one gate call as countable without being able to interpret its result, and attempted to record it. The counter is a mechanical tally, not a count of completed reviews: it can include calls that failed or reviewed nothing, so it can overstate them. Judge every pass on its findings artifact and discount any incomplete or unverified call, whatever the counter says. Normally said once per workspace. It repeats only when its marker cannot be persisted or two hook runs race, so treat a repeat as a marker problem rather than as new information.' +UNVERIFIED_MSG='ℹ A gate call was classified as countable without inspection, and recording it was attempted. Causes with a check and a fix: a pinned-server envelope whose key order or formatting changed — compare the version in .mcp.json with the server actually serving the tools (`claude mcp list`), and pinning it back fixes it; a reworded backgrounding notice — set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code, using 0 to disable auto-backgrounding or a positive value exceeding your longest gate call, needing Claude Code 2.1.212 or newer; a broken awk or sed, which classification requires — check them functionally rather than by version flag, since --version is not POSIX and BSD sed exits nonzero for it on a healthy macOS: `printf "x\\n" | awk "{print}"` must print x and `printf "x\\n" | sed s/x/y/` must print y, each exiting 0. Causes with no user-side fix: a mapped third-party tool whose envelope this hook cannot read — .context/codex-gate.tools names it, and unmapping it removes the gate rather than fixing the envelope; a payload the scan refused as oversized or ambiguous — these two are **not distinguishable from the outside**, since both produce this same message and the same state, so measure the payload size against the 1 Mi-unit ceiling to rule the first in or out and treat the rest as unresolved; and a defect in this hook parser — same situation. For either, keep the payload **locally and access-restricted**: it can contain prompts, absolute paths, review content, session identifiers and unrelated concurrent call data, so strip those before showing it to anyone, and never attach it unsanitized to a report. The list is not exhaustive: unrecognized is the terminal class, so any future unmatched shape lands here too.' +``` -**A3 — the blank grammar and its ordering.** Blank is defined on the **escaped bytes**: the block is blank when it contains nothing but ASCII space and the two-byte sequences `\n`, `\t`, `\r`. A Unicode-escaped space is *not* blank by this rule and does not need to be — **canonical-form validation runs first** (Task 4, Step 4), so any encoding `jq` would normalize has already returned 2 and become `unrecognized` before the blank test is reached. That ordering is the contract, and Step 3 tests it directly. + **Every path that permits a retry also requires the §5 cleanup.** `FAILURE_CTX`, `NORESULT_CTX`, `BG_LONG_CTX` and `BG_SHORT_CTX` all direct or allow a re-run, and CLAUDE.md §5 requires deleting every target findings file and **confirming it is gone** before each call — precisely because a died-part-way call leaves a valid-looking file that no terminator can distinguish from a fresh one. Retrying without it recreates a false clean pass through a door the counter fix does not close. Each of those four `` sections therefore ends with: *"Before re-running, delete the pass's target findings file and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume."* The backgrounded pair keeps that **after** its stop-or-await instruction, in that order: deleting a slot while the original call can still write it is the race those messages exist to prevent. -- [ ] **Step 1: Write the failing tests** + **Oracle for this step, and it is not optional: pipe the ten assignments through `sh -n`.** They are single-quoted shell strings, so **no ASCII apostrophe may appear anywhere inside them** — POSIX shell cannot escape one within single quotes, and `\'` opens an unterminated quote rather than escaping anything. That defect reached this plan twice (`printf \'x\\n\'`, and a `pass's` in the cleanup sentence) and both would have made the shipped hook unparseable. Write inner examples with double quotes, and phrase possessives around the apostrophe. **Same check after embedding into the hook**: `sh -n`, then `shellcheck --shell=sh`. -```sh -c() { CLASSIFY_INPUT="$1" sh "$HOOK_CLASSIFY"; } - -[ "$(c "$(cat "$FIXTURES/shape0-success.json")")" = success ] && pass "success fixture" || fail "success fixture" -[ "$(c "$(cat "$FIXTURES/shape1-fast-fail.json")")" = failure ] && pass "fast-fail fixture" || fail "fast-fail fixture" -[ "$(c "$(cat "$FIXTURES/shape2-executor-timeout.json")")" = failure ] && pass "timeout fixture" || fail "timeout fixture" -[ "$(c "$(cat "$FIXTURES/shape3-backgrounding-notice.json")")" = backgrounded ] && pass "notice fixture" || fail "notice fixture" - -# blank grammar -[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":" \n\t "}]')")" = no-result ] \ - && pass "blank text is no-result" || fail "blank text is no-result" -# ordering: unicode-escaped space fails canonical form FIRST -[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":" "}]')")" = unrecognized ] \ - && pass "unicode space is unrecognized, not blank" || fail "unicode space is unrecognized, not blank" -# reordered failure envelope counts as unrecognized, per the table -[ "$(c "$(payload mcp__codex__exec '[{"type":"text","text":"{\"status\": \"error\", \"success\": false}"}]')")" = unrecognized ] \ - && pass "reordered envelope is unrecognized" || fail "reordered envelope is unrecognized" -# collision: a success whose summary quotes both literals -[ "$(c "$(cat "$FIXTURES/collision-success-quotes-both.json")")" = success ] \ - && pass "collision fixture classifies by its own envelope" || fail "collision fixture classifies by its own envelope" -``` + Six wordings are load-bearing and were changed for stated reasons, not style. **`CODEX_EXECUTION_FAILED` does not mean "the call never started"** — the pinned server uses it generically for abort-before-start, abort-after-start and child execution errors, so telling Claude it never started invites overwriting or colliding with artifacts from a call that did. **The timeout remedy is operator-side** and lives in `systemMessage`; the model gets the retry-same-scope rule and nothing it lacks the authority to do. **`NORESULT_MSG` narrows without concluding** — spec §6 says the tool name cannot distinguish the two causes, so provenance evidence must not become a causal claim. **`UNVERIFIED_MSG` keeps the spec's "no user-side fix"** for the unreadable mapped tool: unmapping removes the gate, which is not a fix, and an earlier draft's reclassification contradicted approved spec §6. **Its `awk`/`sed` check is functional**, because `--version` is not POSIX and BSD `sed` exits nonzero for it. **`UNVERIFIED_CTX` says "classified as countable and attempted to record"**, never "was counted" — the counter write is best-effort, and a categorical claim about recorded state is item 11 exactly. -Create `collision-success-quotes-both.json` and `collision-failure-quotes-true.json` in `fixtures/` by editing a copy of `shape0-success.json` / `shape1-fast-fail.json` so the `summary` field contains both marker literals. These two are **synthetic by necessity** — no real call produces them — and the README says so. +- [ ] **Step 2: Install the locator and matcher** from `.context/plan-drafts/{locate.awk,match.sh}`. -- [ ] **Step 2: Run them to see them fail** + The `awk` program becomes `LOCATE_AWK`, **a single-quoted shell variable — so it must contain no ASCII apostrophe anywhere, comments included.** Two crept into its comments during this cycle (`caller's`, `machine's`) and were removed; one would terminate the quote and leave the hook unparseable. **Verify mechanically after embedding**: `sh -n` on the hook, then `shellcheck --shell=sh`, then one classification test — a syntax error here is not subtle, but it is also not something a reader reliably spots in a 90-line embedded program. `match.sh` installs as ordinary functions, so its apostrophes are fine; the constraint is the quoting, not the file. -Expected: 8 failures — `classify` does not exist. + `locate_result` is the one-line wrapper the classifier calls, and its contract is: feed **`$payload` unchanged** to `LOCATE_AWK` via `awk`, and return `awk`'s status untouched. It must not pre-process the payload, must not consult `jq`, and must not collapse statuses — `0` located, `1` unambiguously nothing there, **anything else** cannot-determine. Collapsing "anything else" to a specific class is how a missing `awk` (127) would become `no-result` instead of the fail-open class. -- [ ] **Step 3: Implement** + Then `strip_ws`, `_token_ends`, `classify_block`, and: ```sh classify() { blk=$(locate_result); rc=$? - [ "$rc" = 1 ] && { printf 'no-result'; return; } - [ "$rc" = 2 ] && { printf 'unrecognized'; return; } - case "$blk" in - 'MCP tool "'*'" is still running after '*) printf 'backgrounded'; return ;; - esac - # blank: only ASCII space and the escapes \n \t \r - case "$(printf '%s' "$blk" | sed 's/\\[ntr]//g; s/ //g')" in - '') printf 'no-result'; return ;; - esac - case "$blk" in - '{'*'\"success\":'[[:space:]]*'true'*) printf 'success'; return ;; - '{'*'\"success\":'[[:space:]]*'false'*) printf 'failure'; return ;; - esac - printf 'unrecognized' + [ "$rc" = 1 ] && { printf 'no-result'; return 0; } + [ "$rc" = 0 ] || { printf 'unrecognized'; return 0; } + classify_block "$blk" } ``` -The `{` prefix plus immediate key is what enforces "immediately-first"; a reordered envelope falls through. - -- [ ] **Step 4: Run to verify they pass, both parser environments** - -Expected: `all passed` in both. - -- [ ] **Step 5: Commit** + `[ "$rc" = 0 ] ||` rather than `[ "$rc" = 2 ] &&`: a missing or failing `awk` exits 127, and every status that is not "located" or "nothing there" must reach the fail-open class rather than fall through to a matcher holding an empty string. Two properties belong in comments because a later edit can silently break them: the locator accumulates records and works in `END`, restoring the newline `awk` stripped, because `RS="\0"` is not portably a record separator; and it prints with `printf "%s"`, so no trailing newline is appended — the caller reads it through command substitution, which strips trailing newlines, harmless only because a JSON string cannot contain a raw newline. -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh plugins/dev-workflow/hooks/fixtures -git commit -m "feat(hooks): classify the gate result into five classes" -``` - ---- +- [ ] **Step 3: Wire the classes to state effects.** For each gate tool: `success|unrecognized` fall through to today's behaviour, with `unrecognized` also calling `note_unverified`; every other class calls `note_discarded` and **falls through to `flush_notes`** — no `exit 0` in the discarded branch, or the message just buffered is never written. -### Task 6: Wire classes to state effects +- [ ] **Step 4: Implement the diagnostic interface and extend `flush_notes`** per the A5 table. `note_unverified` sets a flag and nothing else: the *decision* is there, the *delivery and state* are in the flush, because only the flush knows whether anything was written. -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh:361-397` (the `PostToolUse` branches) -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +**Oracles for Task 5.** Each row is a test label and the thing it must fail on. Where a row says "both emitters", it runs twice — once through the normal runners, once through the jq-free pair — and a wrapper that declares the parameters without using them is not coverage. -**Interfaces:** -- Consumes: `classify` (Task 5). - -- [ ] **Step 1: Write the seeded-preservation tests, both gates** - -Empty-state assertions would pass an implementation that clears earned state, so every case seeds first. +*Classification — ported from `.context/plan-drafts/verify.sh`.* **Port by label, and account for every one.** The corpus is **53 call sites producing 56 assertions** — one `chk` inside a `for` loop covers the four real captures, so a label count and an assertion count are different numbers and neither substitutes for the other. Enumerate with: ```sh -for cls in shape1-fast-fail shape2-executor-timeout shape3-backgrounding-notice; do - reset_all; rev; rev # earn two real Gate-B passes - before_count=$(cat "$count"); before_fresh=$(cat "$fresh"); before_state=$(cat "$state") - run "$(payload mcp__codex__review "$(jq -c '.tool_response' "$FIXTURES/$cls.json")")" >/dev/null - [ "$(cat "$count")" = "$before_count" ] && pass "$cls preserves passCount" || fail "$cls preserves passCount" - [ "$(cat "$fresh")" = "$before_fresh" ] && pass "$cls preserves freshCount" || fail "$cls preserves freshCount" - [ "$(cat "$state")" = "$before_state" ] && pass "$cls preserves fingerprint" || fail "$cls preserves fingerprint" - - reset_all; execp; execp # earn two real Gate-A passes - beforeA=$(cat "$countA") - run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/$cls.json")")" >/dev/null - [ "$(cat "$countA")" = "$beforeA" ] && pass "$cls preserves passCountA" || fail "$cls preserves passCountA" -done +grep -nE '^[[:space:]]*(chk|p_case) "' .context/plan-drafts/verify.sh | sed 's/".*//' ``` -- [ ] **Step 2: Add the success-path discrimination test** + For each of the 53, record *ported* or *excluded, with which of the two exclusions applies*. A group heading cannot show that `non-object element skipped` or an array-walk boundary quietly stopped being required, which is exactly the decision-procedure-replacement failure `AGENTS.md` names. Every ported row must be a payload the hook **routes**: a complete `hook_event_name` and a gate `tool_name`. A bare `{"tool_response":…}` fragment routes to nothing and would report an unidentified class for every row — a table that looks ported and asserts nothing. -`success` and `unrecognized` have identical counter and fingerprint effects, so a counter assertion alone cannot tell them apart. +| Group | Must fail on | +|---|---| +| the four real captures | any drift in the shapes this whole change was built from | +| both collision fixtures | a success quoting `false` classifying as failure, or the reverse. The failure direction is where a mistake produces the false ✓ | +| polarity grammar: compact, tab, space-before-colon, CRLF, reordered, glued token, past the 64 bound | a whitespace form silently reclassifying, and `truely` reading as `true` | +| notice anchor + its four near-misses | the anchor matching text that merely contains the phrase, or missing the real capture | +| every `no-result` shape | any of them counting | +| both decoy directions — `tool_input` quoting the key **before**, a result quoting it **after** | a byte-position heuristic returning | +| **block selection**: a non-text first element followed by a real text block, classified from the text block | an `element [0]` implementation, which passes every `no-result` and capture row while violating spec §3.1's settled first-`text`-element rule | +| a duplicate `type`/`text` in a **preceding non-text** element | the stricter any-examined-element refusal regressing while the selected-element duplicate rows stay green | +| duplicate depth-1 key; duplicate `type`/`text` | last-wins classifying an ambiguous payload | +| the three **walkable-but-invalid** rows — **locator-level, exempt by label from the routing rule below** | a future tightening turning the locator into a validator, or a claim about validation widening again. They are malformed *outer* JSON, so with `jq` they never route and with `grep` they do: requiring them to run through the hook would make them vacuous in one environment and untestable in the other. They stay contract tests over the locator, which is the level their claim is about | +| the length ceiling; the depth cap at 201 openers; and ordinary nesting unaffected | either bound disappearing, **or the cap being set so low it refuses real payloads** — the third row is what makes the second safe to tighten | +| Unicode-escaped marker key | a payload malformed for a *different* reason passing as this test. The located text must contain the **six-byte** `\u0022success\u0022` spelling and **no raw quote** around `success`; an earlier version supplied raw quotes, which made the outer payload malformed and reached `unrecognized` from the locator rather than the matcher. Assert the locator **succeeded** before the matcher's verdict | + +Two groups stay in the drafts and are **not** ported, stated rather than dropped: deliberately unroutable documents (spec §3.3 excludes them), and documents whose routability depends on `jq` (their behaviour is two different things in two environments, neither the classifier's doing). + +*State effects.* + +| Label | Must fail on | +|---|---| +| `// writes no gate-pass state from clean` | an implementation that recomputes and stores the **current** fingerprint over a seeded identical one — byte preservation alone cannot see that. Says **gate-pass** state: a first backgrounded call writes `bgAdvice` by design | +| `// preserves passCount/freshCount/fingerprint/passCountA` | any discarded class touching earned state, on either gate, under either tool-name source, in **both emitters** | +| ` creates no diagnostic marker` / `backgrounded creates its diagnostic marker` | the two state families being conflated | +| `success counts` + `success creates no disclosure marker` | `unrecognized` passing as `success` — their counter and fingerprint effects are identical by design | +| `real review capture is class success with no marker` | a drifted `shape0-success-review` counting through the fail-open terminal class. **Count and fingerprint alone cannot see this** — assert the class | +| `: payload and response slice locate identical bytes` — one per fixture, the **permanent** form of Task 1 Step 7 | a fixture edited without its slice. Class-equivalent drift leaves every classification row green while the README's byte-exact claim about the duplicated representation is false. Both locator statuses must be 0 before the bytes are compared | -```sh -reset_all -rev -[ "$(cat "$count")" = 1 ] && pass "success counts" || fail "success counts" -[ ! -f "$unverified_file" ] && pass "success creates no disclosure marker" || fail "success creates no disclosure marker" -``` +*Markers, both emitters, one label per A5 row.* Each needs a **surgical** fault, and the two blunt approaches both failed: replacing `.context` with a file removes the adoption marker so the hook exits before classifying, and `chmod 500 .context` breaks the counter writes while stdout still succeeds. What separates operations: a **directory** at a marker path fails `printf > f`; a non-writable `.context` fails `rm` while an **existing** file can still be truncated. -- [ ] **Step 3: Run to see them fail** +| Label | Must fail on | +|---|---| +| `on: writes shown, owes nothing` · `off: owes pending, no shown` · `failed write: owes pending` | any of the three flush outcomes writing the wrong marker | +| `shown-write failure keeps the debt` | the debt dying with the marker | +| `pending flushes prefixed` · `flush clears pending` | a carried disclosure losing its `Earlier: ` or its cleanup | +| `pending+unrecognized: not prefixed, ctx exact, shown written, pending cleared, still counts` | the precedence between `note_unverified` and the pending check inverting | +| `shown+pending resolves to shown` · `the next event clears the coexistence` | the retry never happening. Needs a **real** pending file: a directory there is not seen as pending at all | +| `shown means silent` | a spent one-shot re-firing | +| `C2/1` and `C2/2`: counted, no output, neither marker, exit 0 | either direction of the accepted residual being quietly closed **or widened** | +| `long advice writes its marker` · `short form names the variable` · `off/failed write does not burn the bg one-shot` · `bgAdvice write failure repeats the long form` | the one-shot burning on a message nobody saw | +| exit 0 with a directory at **each** marker path, under `sh` **and** `dash` | the special-builtin exit | + +*Fault tolerance of the two load-bearing tools.* Three shapes each — absent, nonzero exit, partial output then failure — through **both** gate tools. + +| Label | Must fail on | +|---|---| +| `: exits 0, review counts, stores a usable fingerprint, discloses; exec counts` | fail-**closed** on pass state while disclosing uncertainty. Input is a real **failure** envelope, the one case where fail-open costs a real count | +| `: exits 0, a success envelope still counts, discloses` | the blank test's empty output reading as blank and turning a genuine success into `no-result` | -Expected: the preservation tests fail — every class still counts today. +*Messages and composition.* -- [ ] **Step 4: Implement** +| Label | Must fail on | +|---|---| +| golden on **both fields** for `failure`, `no-result`, `backgrounded` long and short, the disclosure alone, and one composed pair | a negation, a dropped remedy, a reordered composition. Clause greps catch none of those. Expected values are **literal copies** — a golden reading the hook's own variable agrees with any text the hook emits | +| `: exactly one document` — for **fourteen**: the nine existing emitting branches from Task 4, plus `failure`, `no-result`, `backgrounded` long, `backgrounded` short, and the disclosure alone | zero output as much as two. Count them explicitly: an omitted branch is the dropped-output failure this oracle exists to catch, and an earlier draft said thirteen while listing fourteen sources | +| `: composed ctx/msg exact` | truncation, reordering, duplication or negation of a branch message | +| `: pending cleared after a successful flush` | the debt surviving delivery | +| `silent: flushes the debt alone, no separator` | the wrapper failing on the branch that emits nothing of its own | +| one two-message branch through the **jq-free** emitter, both fields exact, then parsed by a real JSON parser | the fallback escaper mangling the longest, most punctuated string it ever handles | -```sh - "$review_tool") - cls=$(classify) - case "$cls" in - success|unrecognized) ;; # fall through to today's behaviour - *) note_discarded "$cls"; exit 0 ;; # no gate-pass state at all - esac - mkdir -p "$state_dir" 2>/dev/null - # ... existing fingerprint/fresh/count logic unchanged ... - ;; - "$exec_tool") - cls=$(classify) - case "$cls" in - success|unrecognized) mkdir -p "$state_dir" 2>/dev/null; bump_count "$countA_file" ;; - *) note_discarded "$cls" ;; - esac - ;; -``` +**The composition matrix has one sequencing requirement**, and getting it wrong makes five rows fail for a harness reason: the pending disclosure must still be owed when the **observed** event runs. Scenario setup that itself emits will flush and clear the debt first. Keep the off-switch on through setup and remove it immediately before the observed event. -- [ ] **Step 5: Run to verify, both parser environments. Commit.** +**Every exact field comparison must handle the jq-free case the same way.** Without `jq` the fallback extractor returns the field's **escaped** bytes, so a decoded expectation fails on any message containing a quote — and every message here contains quotes. One helper, used by every comparison, not just some. -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "feat(hooks): discarded classes write no gate-pass state" -``` +Commit: `WIP: feat(hooks): classify gate results and stop counting the ones that reviewed nothing`. --- -### Task 7: `failure`, `no-result` and `backgrounded` messages — discharges A4 - -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` — `note_discarded()` -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +### Task 6: The shipped statements this change falsifies — discharges B1, B2, B3 -**A4 — the notice grammar.** The anchor is exactly: the block begins `MCP tool "`, and the segment `" is still running after ` occurs before any newline. **Variable spans, explicitly outside the anchor:** the quoted tool name (`codex/exec`, `codex/review`, or any mapped name), the threshold digits and unit, and the task id. **Near-misses that must NOT match:** a block merely containing the phrase later in its text; a result whose summary quotes the whole notice; a block beginning `MCP tool "` with no `is still running after` segment. +**Files:** `CLAUDE.md`, `README.md`, `commands/workflow-init.md`, `codex-gate.sh`, **`codex-gate.test.sh`**, `AGENTS.md`, `docs/architecture.md`. The hook because B3 includes the unknown-tool message; the last two because Task 1 added a shipped directory and both carry layout trees. -- [ ] **Step 1: Write the failing message tests, including near-misses** +- [ ] **Step 1: Census.** Three greps, run 2026-08-01. The mechanism claim has **five** sites, not the two the B1 list carried — the extra three phrase it differently, which is why the pattern needs three alternations: ```sh -out=$(run "$(payload mcp__codex__review "$(jq -c '.tool_response' "$FIXTURES/shape1-fast-fail.json")")") -printf '%s' "$out" | grep -q 'not counted' && pass "failure says not counted" || fail "failure says not counted" - -out=$(rev_noresult) -printf '%s' "$out" | grep -q 'no tool result' && pass "no-result names its cause" || fail "no-result names its cause" -printf '%s' "$out" | grep -q 'mapped tool' && pass "no-result names BOTH causes" || fail "no-result names BOTH causes" - -# near-miss: notice text quoted inside a real envelope must NOT be backgrounded -nm='[{"type":"text","text":"{\"success\": false, \"summary\": \"MCP tool \\\"codex/exec\\\" is still running after 120s\"}"}]' -[ "$(c "$(payload mcp__codex__exec "$nm")")" = failure ] && pass "quoted notice is not backgrounded" || fail "quoted notice is not backgrounded" +# `git grep` over TRACKED files, not recursive grep: `.mcp/` holds generated cache copies +# of the hook (AGENTS.md defines it as generated state), and a recursive walk edits or +# counts files nobody ships. An earlier draft said this in prose and left the commands +# recursive — the prose is not the command. +X='source-files/|docs/superpowers/' +git grep -niE 'key(s|ed) on (the )?tool.?name|key on those|counts passes by tool|never (sees|inspects|reads)' -- '*.md' '*.sh' | grep -vE "$X" +git grep -niE 'accurate' -- '*.md' '*.sh' | grep -vE "$X" +git grep -niE 'execTool|reviewTool|codex-gate\.tools' -- '*.md' '*.sh' | grep -vE "$X" ``` -- [ ] **Step 2: Run to see them fail. Step 3: Implement `note_discarded`.** +| Site | Phrasing | Disposition | +|---|---|---| +| `CLAUDE.md:168` | "keyed on tool name, and never sees the file" | Step 2 | +| `commands/workflow-init.md:347` | same sentence, inline template | Step 2 | +| `commands/workflow-init.md:250` | "The hook counts passes by TOOL NAME" | Step 2 | +| `README.md:59` | "the gates and their pass counters key on those two tool names" | Step 2 | +| `AGENTS.md:70` | "(the gates key on those two tool names)" | **read and decide** — it describes which MCP tools back the gates, which classification does not change | -```sh -note_discarded() { - case "$1" in - failure) - emit "This Codex call reported failure, so it was not counted as a gate pass and no review fingerprint was stored. An incomplete pass does not count toward the floor." \ - "⚠ Codex call failed — not counted as a gate pass" ;; - no-result) - emit "The payload carried no tool result, so this call was not counted as a gate pass. Two causes are possible and the tool name alone cannot tell them apart: a hooks-API payload contract change (check your Claude Code version and report it — the pinned server cannot produce this shape), or a mapped tool returning empty or non-text content. Check .context/codex-gate.tools for an active mapping and .mcp.json for the effective server." \ - "⚠ Gate call returned no readable result — not counted" ;; - backgrounded) note_backgrounded ;; - esac -} -``` +**The greps decide**, not the list — but read the hits, do not count them. Verified 2026-08-02: grep 1 returns **seven lines for five sites**, because the `CLAUDE.md` and `workflow-init.md` sentences each wrap across two lines (`:168`/`:169`, `:347`/`:348`). A count comparison would report a phantom two-site discrepancy every time. -- [ ] **Step 4: Implement `note_backgrounded` with the one-shot advice** +**Grep 2 has two hits the B2 list does not carry, and they are in the suite:** `codex-gate.test.sh:296` (the section comment *"Full state machine keeps running while off (so re-enable is accurate)"*) and `:306` (the assertion label *"re-enable sees accurate state"*). Both assert the exact overclaim §5.2 retires. Reword both — comment and label — to *"same counting semantics as gate-on, not evidence of review"*, and add `codex-gate.test.sh` to this task's staged paths. Test-suite hits are assertions **about** the strings and change with them; that is a reason to disposition them, not to exclude them. -```sh -note_backgrounded() { - if [ -f "$bg_advice_file" ]; then - emit "Gate pass discarded — the call was backgrounded and its result never reached the hook. Not counted. See CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS." \ - "⚠ Gate pass discarded (backgrounded) — not counted" - else - if emit "This gate pass was discarded, not counted: the call was moved to the background at the auto-background threshold (120 s by default), so its result never reached this hook. This is a setup gap, not a failed review." \ - "⚠ Gate pass discarded: set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from, then restart it (it is read at process start, so exporting it inside a tool shell has no effect). 0 disables auto-backgrounding; a positive value must exceed your longest gate call. Requires Claude Code >= 2.1.212."; then - mkdir -p "$state_dir" 2>/dev/null - { : > "$bg_advice_file"; } 2>/dev/null || true - fi - fi -} -``` +Exclude generated state explicitly: `.mcp/` holds cache copies of the hook (`AGENTS.md` defines it as generated), and a recursive grep that reaches them sends the census into files nobody ships. `git grep` over tracked files is the simpler guard and is what these commands should use. -The marker is written **only** on `emit` returning 0 — suppressed or failed writes leave the one-shot unspent. +- [ ] **Step 2: B1 — correct one clause, keep the other.** *"…keyed on tool name, and never sees the file"* becomes *"…keyed on tool name **and on the result envelope**, and still never sees the file."* **"Never sees the file" stays** — it is about the *findings file*, and discounting an incomplete pass whose findings file is missing remains instruction-backed. -- [ ] **Step 5: Test the one-shot and its non-burning. Step 6: Commit.** + *"A failed review therefore looks like a successful tool call and increments the counter"* becomes: -```sh -reset_all; run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/shape3-backgrounding-notice.json")")" >/dev/null -: > "$off_file" -out=$(run "$(payload mcp__codex__exec "$(jq -c '.tool_response' "$FIXTURES/shape3-backgrounding-notice.json")")") -[ -z "$out" ] && pass "off suppresses the short form" || fail "off suppresses the short form" -``` +> A failed review therefore still looks like a successful *tool call* — but as of 0.8.0 the hook reads the result of gate calls it can route, and withholds the count for three **recognized** shapes: an envelope whose **first** property is `success: false`, the harness backgrounding notice **in the wording it currently uses**, and a result from which no usable text can be obtained. Every other routed gate call counts, including any located text the hook cannot interpret — a reordered envelope, a reworded notice, an unknown third-party shape — which counts **with** a disclosure that is attempted and normally shown once per workspace, but can be lost or repeated when its marker cannot be persisted. So the counter is closer to the truth than it was and is still not evidence: **discount every incomplete pass regardless of what the counter says**, because classification cannot see whether the findings file was written. -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "feat(hooks): report discarded passes with cause and fix" -``` + **Six precisions there are load-bearing** and none may be dropped when it is edited: **first property**, not "contains"; **the current wording**, not "a backgrounding notice" (a reworded one is counted — C1); **routed** gate calls, since an unroutable payload touches no state at all; **"located text it cannot interpret" ≠ "no text at all"** — the first is `unrecognized` and counts (decision 2), the second is `no-result` and does not (decision 3), and collapsing them contradicts a settled decision either way; **"no usable text"**, not "absent", since `no-result` also covers null, empty, non-array, non-object, non-text and blank; and the disclosure is **attempted and normally once**, not guaranteed (C2, C4, item 11). ---- + `commands/workflow-init.md:250` → *"counts passes by tool name and by result envelope"*. `README.md:59` → *"the gates key on those two tool names, and the pass counters additionally skip routed calls whose result the hook reads as failed, backgrounded, or yielding no usable text — a result it can read but not interpret still counts, and normally says so once."* -### Task 8: The `unrecognized` disclosure and pending state — discharges A5 +- [ ] **Step 3: B2 — retire "accurate."** `README.md:97` and `commands/workflow-init.md:1038` take spec §5.2's contract: opt-out preserves the same counting semantics as gate-on, and the counters are never evidence a review happened. -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` +- [ ] **Step 4: B3 — every mapping instruction.** Each states that a mapped name must lie in `mcp__codex__*` and that the remedy is registering the server as `codex`. **`commands/workflow-init.md:74` and `:157` offer renaming the project entry *away* from `codex`** — producing exactly the unreachable configuration decision 1 describes. Remove that remedy unless it also re-registers the effective server as `codex`. -**A5 — the complete marker table.** Three files: `codex-gate.bgAdvice`, `codex-gate.unverified` (shown), `codex-gate.unverifiedPending` (owed). They are independent; `bgAdvice` has no interaction with the other two, which is why its lifecycle is a single row rather than a matrix. + The unknown-tool message is a shipped prompt, so its replacement is literal. Add the target-model prefix the other four hook prompts carry, then after *"…genuinely has two tools that separate reviewing TEXT from reviewing a DIFF"*, insert: -| `unverified` | `unverifiedPending` | Event | Result | -|---|---|---|---| -| absent | absent | `unrecognized`, emit returns 0 | write `unverified` | -| absent | absent | `unrecognized`, emit returns 1 (off) | write `unverifiedPending` | -| absent | absent | `unrecognized`, emit returns 2 (write failed) | write `unverifiedPending` | -| absent | present | any unsuppressed event | emit disclosure; on 0 → write `unverified`, delete pending; else retain pending | -| absent | present | `unverified` write fails after successful emit | **retain pending** (duplicate beats loss) | -| present | absent | `unrecognized` again | nothing emitted, nothing written | -| present | present | any | treat as shown; delete pending. Reachable only if a delete failed earlier | -| any | any | marker write fails | proceed, exit 0 — **C2 applies** | +> **A mapped name must also lie in the `mcp__codex__*` namespace.** This hook is invoked by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming a tool outside it never fires and the gate stays silent — the mapping looks applied and does nothing. -- [ ] **Step 1: Write the state-transition tests, one per row** + And to its `systemMessage`, because registering a server is an operator action: -```sh -reset_all -run "$(payload mcp__codex__review "$(unrec)")" >/dev/null -[ -f "$unverified_file" ] && pass "gate-on unrecognized writes shown" || fail "gate-on unrecognized writes shown" - -reset_all; : > "$off_file" -run "$(payload mcp__codex__review "$(unrec)")" >/dev/null -[ -f "$pending_file" ] && pass "gate-off unrecognized writes pending" || fail "gate-off unrecognized writes pending" -[ "$(cat "$count")" = 1 ] && pass "unrecognized still counts while off" || fail "unrecognized still counts while off" - -rm -f "$off_file"; out=$(rev) -printf '%s' "$out" | grep -q 'cannot verify' && pass "pending flushes on re-enable" || fail "pending flushes on re-enable" -[ ! -f "$pending_file" ] && pass "flush clears pending" || fail "flush clears pending" -``` +> A mapped tool name must start with `mcp__codex__`, or the gate never fires. Register the server under the name `codex` to place its tools there. -- [ ] **Step 2: Run to see them fail. Step 3: Implement.** +- [ ] **Step 5: The layout trees.** Add `hooks/fixtures/` to `AGENTS.md` § Architecture and `docs/architecture.md`. Two greps, because the Don'ts require both: `grep -rn 'codex-gate' AGENTS.md docs/architecture.md MANIFEST.md README.md`, and the mandated manifest-claim census `grep -rniE 'declare[sd]?|convention[- ]load' --include='*.md' . | grep -v source-files/`. **Then read `plugins/dev-workflow/.claude-plugin/plugin.json` in this same change** and check every hit against it — three sites once shipped the claim that the manifest declares `hooks` when it declares nothing, and no mechanical check can tell whether a sentence about a manifest is true. -```sh -note_unverified() { - [ -f "$unverified_file" ] && return 0 - if emit "$UNVERIFIED_CTX" "$UNVERIFIED_MSG"; then - mkdir -p "$state_dir" 2>/dev/null - if { : > "$unverified_file"; } 2>/dev/null; then - rm -f "$pending_file" 2>/dev/null - fi # else: pending retained on purpose - else - mkdir -p "$state_dir" 2>/dev/null - { : > "$pending_file"; } 2>/dev/null || true - fi -} -``` +- [ ] **Step 6: README setup for the environment variable** (spec §9, a story acceptance criterion). `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` with **every settled clause**: requires Claude Code ≥ 2.1.212; set in the environment Claude Code is **launched from**, so a running session must be restarted; `0` disables auto-backgrounding; a positive value must **exceed** the longest expected gate call. And **both** outcomes without it — while the notice keeps its current wording the pass is *discarded*, and if that harness prose changes the call is *counted* with a disclosure. That second outcome is C1, and it is why this setting is the primary defence rather than the anchor; a Setup section promising only "discarded" hides the residual where the operator could still prevent it. **`CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` is not documented** — it exists in the 2.1.220 string table but was never exercised, and naming the wrong one of two is this repo's docs-drift class. -`$UNVERIFIED_CTX` enumerates the causes with their remedies, per spec §6 — a changed pinned-server envelope (check the version in `.mcp.json`), a reworded backgrounding notice (the `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` guidance), a mapped third-party envelope (**no user-side fix**), and a hook parser defect (**no operator fix**) — and states that the list is not exhaustive because `unrecognized` is the terminal class. +- [ ] **Step 7: Golden assertions for the two edited prompts**, and update the unknown-tool golden in **both** places it appears — the branch assertion and the composition matrix — or the composed assertion fails on a change the branch assertion accepted. -- [ ] **Step 4: Verify, both parser environments. Commit.** +- [ ] **Step 8: All 12 prompt-standards items** for every string added or changed in Tasks 5 and 6. -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "feat(hooks): disclose uninspected passes once per workspace" -``` +Commit: `WIP: docs: describe result classification where the old mechanism was taught`. --- -### Task 9: Composition — discharges A6, A7 - -**Files:** -- Modify: `plugins/dev-workflow/hooks/codex-gate.sh` -- Test: `plugins/dev-workflow/hooks/codex-gate.test.sh` - -**A6 — every emit branch.** A pending disclosure can coincide with: the Gate-B STOP/stale/satisfied reminders, the Gate-A floor reminder, the WIP note, the docs-only note, the unknown-tool note, a `failure`/`no-result`/`backgrounded` note, **and events that emit nothing at all**. The last is why the flush is a wrapper rather than a call inside each branch. - -**A7 — separator and encoding.** The two bodies join with `" — "` (space, em dash, space) in `additionalContext` and `" "` in `systemMessage`. **No newline**, because the `jq`-free emitter escapes only backslash and quote, and a literal newline would produce an invalid JSON document. The disclosure is prefixed `Earlier: ` so a prior-event disclosure cannot read as a statement about the current call. +### Task 7: Changelog, validation evidence, and Gate B -- [ ] **Step 1: Write the collision test for a silent event** +- [ ] **Step 0: Define the evidence file, before anything reads it.** ```sh -reset_all; : > "$off_file" -run "$(payload mcp__codex__review "$(unrec)")" >/dev/null # owes a disclosure -rm -f "$off_file" -out=$(run '{"hook_event_name":"PostToolUse","tool_name":"Edit","tool_input":{"file_path":"a.ts"}}') -printf '%s' "$out" | grep -q 'Earlier:' && pass "silent event still flushes pending" || fail "silent event still flushes pending" -printf '%s' "$out" | jq -e . >/dev/null 2>&1 && pass "flushed output is one valid JSON document" || fail "flushed output is one valid JSON document" +EVIDENCE="$(git rev-parse --show-toplevel)/.context/evidence-0.8.0.md" # .context/ is ignored +: > "$EVIDENCE" ``` -- [ ] **Step 2: Run to see it fail. Step 3: Implement the single-emit wrapper.** + Steps 2–5 each **append** their result to it: the counterfactual's verbatim `FAIL` lines, the named-verification table, the `awk`-portability statement, and the rollback outcome. Before every commit or amend that embeds it, require it **readable and non-empty**, and re-read it — a fix changes the diff even when the profile sits still. -Route every emit through one function that prepends a pending disclosure when one is owed, and — for an invocation that would otherwise emit nothing — flushes it alone before the hook exits. + This step exists because `$EVIDENCE` was referenced by three commit commands and **assigned nowhere**; the scratch dry run that "passed" had defined the variable itself, so it proved the `git` mechanics and never tested the plan's own. Keep it out of the temp directories Step 2 cleans up. -- [ ] **Step 4: Assert exactly one JSON document per invocation across every branch** +- [ ] **Step 1: The CHANGELOG entry.** `0.8.0`, newest first. Names the behaviour change **and C1–C4 by name**. It lands here, not in Task 1, so no intermediate commit carries a release note for behaviour that does not exist. -```sh -for scenario in gateb_stop gatea_floor wip docsonly unknowntool failure noresult backgrounded silent; do - out=$(run_scenario "$scenario") - [ "$(printf '%s' "$out" | grep -c '^{')" -le 1 ] && pass "$scenario emits at most one document" || fail "$scenario emits at most one document" -done -``` - -- [ ] **Step 5: Commit** - -```sh -git add plugins/dev-workflow/hooks/codex-gate.sh plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "feat(hooks): compose an owed disclosure into a single emit" -``` - ---- +- [ ] **Step 2: The `+check` counterfactual.** Materialize the pre-change hook from `$BASE` beside the **new** suite and fixtures in a temp directory. `git stash` cannot do this — Tasks 3–5 already committed the changed hook. -### Task 10: The shipped statements this change falsifies — discharges B1, B2, B3 + **Requirements, each of which an earlier draft got wrong:** the evidence file lives **outside** the temp directory, or the cleanup deletes it before the assertions read it; every guard (`git show`, `cmp`, empty blob) **exits nonzero**, since a bare `echo` continues against a wrong or empty hook; the cleanup is a function with signal handlers that **exit**, and it removes the evidence files too; and **status and completion are checked, not just `grep`** — piping a suite through `grep` hides a crash, a `set -u` abort or a truncated run, so the new suite must exit 0 **and** print its completion marker, and the old one must terminate normally. -**Files:** -- Modify: `CLAUDE.md` §5; `README.md`; `plugins/dev-workflow/commands/workflow-init.md` + Then assert **exact labels**, not "something failed". The four, named here so the counterfactual is auditable against this plan rather than against whatever the suite happened to produce — only the mechanical runner suffix is filled in at execution: -**This task is where B becomes work.** Nothing in it is edited before this point. + 1. `failure/default/ preserves passCount` + 2. `timeout/default/ preserves passCount` + 3. `failure/default/ writes no gate-pass state from clean` + 4. `backgrounded/default/ writes no gate-pass state from clean` -- [ ] **Step 1: Find every occurrence rather than trusting the list** + Each must be **present** in the base-hook run's failures and **absent** from the new run's. Those four are chosen because each fails for the change's own reason — the old hook counts a discarded class — rather than for a harness difference. **An observation to record, not an assertion to make**: the matching lines go verbatim into the evidence entry. -```sh -grep -rn "keys on tool name\|never inspects\|accurate" --include='*.md' . | grep -v 'source-files/\|docs/superpowers/' -grep -rn "execTool\|codex-gate.tools" --include='*.md' . | grep -v 'source-files/\|docs/superpowers/' -``` - -The pass-8 list is a floor, not a census — invariant: the greps decide. - -- [ ] **Step 2: B1 — describe classification, keep the artifact-validation instruction** +- [ ] **Step 3: The named verification** (the story's profile requires it — re-read the header). -In `CLAUDE.md` §5 and the inline CLAUDE template: replace "the hook counts on tool name and never inspects the result" with what the classifier does. **Keep** the rule that an incomplete pass is discounted regardless of the counter — classification does not cover a `success` call whose findings file is missing, and that remains instruction-backed. + **Two parts, and the second is not optional.** Replaying captured payloads through the changed hook establishes the classification; it does **not** satisfy story criterion 10, which asks for the probe methodology re-run against the changed hook with the variable **absent and set**. An earlier draft substituted pre-change captures and said so, which is a live acceptance criterion silently weakened. Either run the live probe in an isolated profile for both variable states, or **obtain and record a human amendment to the story criterion before Gate B**. -- [ ] **Step 3: B2 — retire "accurate"** + For the replay part, every row is guarded: fixture-read status, `sed` substitution status, the routed tool name, the **exact class**, an explicit expected value per row, and the exact row count — failing the step on any mismatch. Unguarded, a missing fixture or failed `sed` supplies an empty payload to an always-zero advisory hook and still prints the expected `absent` counter, so most of the evidence can be false while looking correct. The success row uses **`shape0-success-review`**, not `shape0-success`: the latter is an `exec` capture whose entire result was the word `ok`, and calling it "a genuine pass" would claim Gate-B continuity for content no reviewer saw. -Three sites take spec §5.2's contract: opt-out preserves the same counting semantics as gate-on, and the counters are never evidence a review happened. +- [ ] **Step 4: `awk` portability evidence.** The local battery uses one `awk`; CI's `ubuntu-*` runs `mawk`. **Bind the CI result to the exact reviewed tree** — a run from before the Gate-B fixes does not cover the final amended bytes — and re-run it after any classifier or locator fix. The work is on `main`, so name the branch or PR route used rather than "after pushing", which reads as pushing unreviewed WIP history to `main`. If CI has not run against the final tree, **say so in the evidence entry** rather than citing 56/56 as portability. -- [ ] **Step 4: B3 — every mapping instruction** +- [ ] **Step 5: The rollback, verified.** `.context/codex-gate.off` is **not** a rollback — it suppresses messages while classification and state tracking keep running, so a workspace whose passes are being discarded stays stuck and goes quiet about it. -Each states that a mapped name must lie in `mcp__codex__*` and that the remedy is registering the server as `codex`. **Remove the preflight remedy that renames the server away from `codex`** unless it also re-registers the effective server there — as written it produces exactly the unreachable configuration. + The real rollback is the version-keyed cache path invariant 12 exists for. Verify three things, each a hard stop: the prior hook is **present**; it matches the **released bytes**, resolved as the **newest commit whose manifest still contains 0.7.1** — not the commit that *set* it, and not `git log -S`. Verified 2026-08-02 in a scratch repo: `-S` finds commits where the string's count *changed*, so it resolves to the 0.8.0 bump that **removed** 0.7.1 and yields the new hook's bytes — the check would then report a mismatch on a correct cache. "The commit that set it" is wrong too, because a later docs commit can change the hook while the version sits still, and those are the bytes that shipped. Walk `git rev-list HEAD` and take the first commit whose `plugin.json` matches `"version"[[:space:]]*:[[:space:]]*"0\.7\.1"`. A tag would be simpler, and this repo has **none** — `git tag -l` is empty — so creating one is a prerequisite to record, not to assume; and it **behaves** as 0.7.1 did, by counting a failure envelope in a disposable adopted repo. Make the probe's failure branch exit nonzero **and** check the subshell's status outside it — an `echo "STOP"` inside a subshell lets the release continue past a failed check. -- [ ] **Step 5: Golden assertions for the two prompts** + **Side-by-side bytes are not an operable rollback.** Record and verify how an operator actually *activates* 0.7.1 in this environment — the cache path is an implementation detail and `claude plugin marketplace add` has no version syntax — and confirm the active hook's bytes afterwards. If no verified switch path exists, stop the release rather than shipping a rollback story nobody has run. -The unknown-tool hook message and the scaffolded CLAUDE text are prompts. Pin both with exact-match assertions, following the suite's existing `matches exactly` tests. + **Then put the candidate back, and prove it.** This drill ends with 0.7.1 *active*. Running it before Gate B and not reversing it means every Gate-B pass executes under the old hook — which counts failed calls, so the pass accounting the review depends on is the accounting this change exists to fix. Either run the whole drill in an isolated profile, or reactivate 0.8.0, **byte-verify the active hook**, and reload the session before the first Gate-B call. Verify, do not assume: this is the one step whose failure is invisible until the review is already worthless. -- [ ] **Step 6: All 12 prompt-standards items** + **State what a rollback costs, not only what it leaves behind.** Reverting to 0.7.1 **restores the original defect**: failed, timed-out and backgrounded calls count as gate passes again. It also restores the `dash` special-builtin exit — on Linux, an unwritable `.context/` makes 0.7.1's hook exit 2, violating invariant 1. So rollback is right for "0.8.0 discards passes it should count", and wrong for anything else; a CHANGELOG that mentions only the leftover markers lets an operator roll back believing it merely removes classification. Also record what it does **not** undo: the three diagnostic markers stay in `.context/`, and 0.7.1 ignores them. -Review every string added or changed in Tasks 7–10 against `docs/prompt-standards.md` — all 12, not item 10 alone. +- [ ] **Step 6: Squash into the reviewed snapshot.** Before the battery and before Gate B — the numbering matters, because an earlier draft ran the post-squash battery in a step that came *before* the squash existed. -- [ ] **Step 7: Battery and commit** + **Stage Task 7's own edit first.** Step 1 modified `CHANGELOG.md` and nothing has committed it; `reset --soft` preserves an unstaged edit but the commit that follows will not contain it, so the release note invariant 12 requires would sit outside the reviewed range while the path audit — which reads commits — cannot see it. ```sh -git add CLAUDE.md README.md plugins/dev-workflow/commands/workflow-init.md plugins/dev-workflow/hooks/codex-gate.test.sh -git commit -m "docs: describe result classification where the old mechanism was taught" +git merge-base --is-ancestor "$BASE" HEAD || { echo "STOP: BASE is not an ancestor"; exit 1; } +git log --oneline "$BASE"..HEAD # read it: every commit must be one of THIS plan's WIP snapshots +git diff --name-only "$BASE"..HEAD # read it: every path must be one this plan named +git status --porcelain # read it: CHANGELOG.md must appear, and nothing unexpected +git add plugins/dev-workflow/CHANGELOG.md +git reset --soft "$BASE" +git commit -m "WIP: gate-pass result classification (0.8.0)" -m "$(cat "$EVIDENCE")" +git show --stat --format=%B HEAD # read it: every named path present, evidence body present ``` ---- + **Two `-m` arguments, not `-m` with `-F`** — `git` rejects that combination outright (*"options '-m' and '-F' cannot be used together"*), so the earlier form could not create the snapshot at all. The first `-m` is the subject and carries the `WIP:` prefix `is_wip_commit` looks for; the second is the evidence body. + + The ancestry check and the reads are not ceremony: an unconditional `reset --soft` folds **every** commit since `$BASE` into the snapshot, so a concurrent or unrelated commit gets rewritten into this change, and re-running the step widens the range again. -### Task 11: Version, changelog, and the named verification +- [ ] **Step 7: Full battery on the squashed tree**, including `sh scripts/check-version-bump.sh "$BASE"` — now that the commit exists — with that exact command recorded. Passing `main` on this checkout compares HEAD with itself and proves nothing. -**Files:** -- Modify: `plugins/dev-workflow/.claude-plugin/plugin.json`, `plugins/dev-workflow/CHANGELOG.md` +- [ ] **Step 8: Gate B.** -- [ ] **Step 1: Bump the manifest version** (invariant 12 — a plugin change without one fails CI). + **Squashing happens before the clean pass, never after.** A `reset --soft` moves HEAD, changing `git diff HEAD` and the effective index — two of the three inputs to `tree_hash` (invariant 3) — so squashing after a clean pass invalidates the fingerprint that pass recorded and correctly fires Gate B again, spending the passes rather than preserving them. -- [ ] **Step 2: CHANGELOG entry**, newest first, naming the behaviour change and the three accepted residuals **C1–C3** by name. + **`baseSha` is `$BASE`, passed literally.** Not a live merge-base: this work is on `main`, where `git merge-base main HEAD` returns HEAD and Gate B would receive an **empty range** and could return clean having seen nothing. -- [ ] **Step 3: The `+check` counterfactual** + Minimum three passes, findings to file, clean final pass. -Run the `failure`-class tests against the **pre-change** hook and record that they fail: + **Every amend rebuilds subject *and* body, in one command.** A bare `git commit --amend -m "WIP: …"` replaces the whole message and **erases the evidence body**; `--amend --no-edit` preserves it but carries no `-m`, and `is_wip_commit` greps the command for `-m … wip` — verified — so the hook reads it as a real cycle-closing commit and **resets the counters mid-cycle**. Only the two-`-m` form satisfies both: ```sh -git stash -sh plugins/dev-workflow/hooks/codex-gate.test.sh 2>&1 | grep -E 'FAIL.*(fast-fail|timeout|preserves)' -git stash pop +# after each fix: inspect, stage ONLY the fix paths, audit, revalidate evidence, then amend. +# A bare `--amend` commits nothing new — the fix stays unstaged, the next reviewer reads the +# old range, and the fingerprint sees a worktree the commit does not contain. +git status --porcelain # read it: only the paths the fix touched +git add plugins/dev-workflow/hooks/codex-gate.sh # <- replace with the exact paths THIS fix touched +git diff --cached --name-only # read it: nothing outside that set +git commit --amend -m "WIP: gate-pass result classification (0.8.0)" -m "$(cat "$EVIDENCE")" +git status --porcelain # read it: MUST be empty before the next review call +# closing, after the clean pass — same shape, real subject: +git commit --amend -m "feat(hooks): classify gate results and stop counting the ones that reviewed nothing" -m "$(cat "$EVIDENCE")" ``` -This is an observation to record, not an assertion to make. + **The evidence entry is in the body from the first snapshot**, revalidated and rewritten into `$EVIDENCE` after every fix — a fix changes the diff even when the profile sits still. Every Gate-B call carries the story path and that entry quoted verbatim. -- [ ] **Step 4: The named verification** (the story's profile requires it) + The standing lens applies with unusual force: this change edits `CLAUDE.md` §5 itself, so ask **which existing statements this diff falsifies** — including in files it does not touch. -Re-run the probe methodology against the changed hook, recording the counter and fingerprint reading for each: both failure envelopes; `backgrounded` with the variable absent; `backgrounded` prevented with the variable set; one genuine pass that still counts. Method is in `.context/probe-payloads/INDEX.md`. +--- -- [ ] **Step 5: Full battery, then Gate B** +## Findings that move to Gate B -Run the `quality` row of `AGENTS.md` § Commands. Then CLAUDE.md §5 Gate B: WIP commit, `mcp__codex__review` against its parent, minimum three passes, findings to file, clean final pass, evidence entry in the closing commit body. +Inherited, not dropped — the way this plan inherited spec §11's deferred contracts. Each is a harness-mechanics defect found at Gate-A pass 6, in shell that no longer lives in this document. Gate B reviews the real implementation and must confirm each: ---- +1. **Expected-message constants** — every `*_EXPECTED` name referenced must be assigned, with the inventory checked against the required set rather than only against names already referenced. +2. **The dual-emitter marker loop** — every row must actually run through both runner pairs; a wrapper that declares parameters and leaves the blocks calling the direct runners is not coverage. +3. **The raw-hook runner in the C2 rows** — a `PATH=…` scalar cannot be expanded as a command prefix; use runner functions that perform the redirection and return the hook's status. +4. **The three `sed` fault shims** and the three `awk` ones must exist, and each must be self-verified in both directions before an assertion depends on it. +5. **The jq-free field comparison** must be routed through one helper used by *every* exact comparison, not only `golden()`. +6. **The skip branches** must skip their dependent assertions, not merely print `skip -` and fall through. ## Self-Review -**Spec coverage:** §3.1 → Task 4; §3.2–3.3 → Task 5; §4 → Task 7 (A4) and C1; §5.1 → Task 6; §5.2 → Task 8 (A5); §6 → Tasks 7–9; §7.1 → Task 1; §7.2 → Task 2; §7.3 → Tasks 4–9; §7.4 → Task 11; §9 → Tasks 10–11; §11 → A1–A7 as mapped. +**Spec coverage:** §3.1 → Task 5 (A1, A2); §3.2–3.3 → Task 5; §4 → Task 5 (A4) and C1; §5.1 → Task 5 Step 3; §5.2 → Task 5 Step 4 (A5); §6 → Tasks 4 and 5; §7.1 → Task 1; §7.2 → Task 2; §7.3 → Task 5's oracle tables; §7.4 → Task 7; §9 → Task 6; §11 → A1–A7 as mapped. -**Placeholders:** none. Every code step carries runnable content. Task 10's steps are edits to prose whose exact target text is found by the greps in its Step 1 rather than quoted here, because quoting it would create a fourth copy of the sentences this change exists to correct. +**Placeholders:** the harness shell, deliberately and by scope (see the note at the top and the Gate-B list). Not placeholders: the five message pairs, the class and marker tables, the contracts, every oracle, and Task 7's procedures — those are what Gate A can judge. Task 6's prose *targets* are located by line and grep rather than quoted, because quoting them would create a fourth copy of the sentences this change exists to correct. -**Type consistency:** `locate_result` (0/1/2) is consumed only by `classify`; `classify`'s five strings are consumed only by the Task 6 branches; `emit`'s 0/1/2 is consumed by `note_backgrounded` and `note_unverified`. `$FIXTURES`, `payload()`, `succ()` are defined in Tasks 1–2 and used unchanged afterwards. +**What no reader should infer.** The scan is a locator, not a validator: string-boundary tracking is what makes the walk trustworthy, a walkable-invalid document is walked past (three fixtures pin that), and its two bounds are backstops in `awk` length units and nesting depth, not contracts — neither bounds memory. The marker table describes sequential behaviour only (C4). The `+check` counterfactual shows four named tests depend on the new code, not that the old hook was wrong in every way the new one is right. 56/56 local is shell coverage, not `awk`-implementation coverage. And the oracles state what each test must *fail on*; whether the shell satisfying them is correct is a Gate-B question, which is the point of moving it there. diff --git a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md index 7c8559f..3f98fbb 100644 --- a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md +++ b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md @@ -63,41 +63,44 @@ exactly one of five classes. tool may legitimately return an image or other block first, and indexing blindly would miss the result or feed a non-text block to the matcher. -**Locating is where the two environments differ; matching is not.** Both paths produce the -**same representation** — the block's text in its **escaped** JSON encoding — and hand it to -one matcher (§3.2). There is no decoded-vs-escaped equivalence to maintain, no parity rule -over matching, and no second implementation of the classification contract. - -- **With `jq`:** locate structurally, then **re-encode to the escaped form**. The selector - must implement the structural contract rather than assume it: gate on `tool_response` - being an **array**, skip elements that are not objects, take the first whose `type` is - exactly `text`, and require its `text` to be a **string** — a bare - `.tool_response[]? | select(.type=="text") | .text` does none of these (`[]?` iterates an - object's *values*, `select` errors on a non-object element, and `.text` goes untyped). -- **Without `jq`:** an escape-aware scan locates the same block; what it yields is already - the escaped form. +**One locator, whether or not `jq` is installed.** An escape-aware scan walks the payload's +own bytes, locates the block, and yields its text in its **escaped** JSON encoding — the +single representation the matcher (§3.2) consumes. It must implement the structural contract +rather than assume it: `tool_response` must be an **array**, non-object elements are skipped, +the first element whose `type` is exactly `text` wins, and its `text` must be a **string**. +`jq` plays no part in classification; it remains the emitter's JSON writer and nothing more. +So there is no decoded-vs-escaped equivalence to maintain, no parity rule, no environment +that classifies differently, and no second implementation of anything. + +> **Amended at the plan's Gate A, pass 1 (2026-08-01).** This section previously specified +> **two** locators — a structural `jq` selector and a `jq`-free scan — reconciled by +> re-encoding the `jq` result to escaped form and requiring that byte sequence to occur +> **exactly once** in the payload and **within the located `tool_response` span**. Both +> conditions, the re-encoding step, and the parity apparatus in §7.3 are deleted with the +> second path rather than fenced off, because text that reconciles two things when only one +> exists can do nothing but drift. The in-span check was the trigger: `jq` reports no byte +> offsets, so computing that span needed the scan anyway, which left the `jq` path as a +> component whose only reachable effect was downgrading an agreed result to `unrecognized`. +> Six of the plan's forty pass-1 findings lived in that machinery. Note this is the +> **second** collapse of a dual-path design in this cycle — §3.2's two matchers went the +> same way at the spec's own Gate A — and both times the artifact came out smaller and +> safer. What the deleted checks defended against, a hijacked anchor or an encoding the two +> paths would read differently, is now structurally unreachable rather than tested for. **Byte-position heuristics are unsafe in both directions**, so neither is used: a greedy strip anchors on the *last* `tool_response` match, which a result quoting the key can hijack; a first-match strip anchors on the *earliest*, which `tool_input` can hijack, and -this repo's own gate prompts quote payload text. - -**The matcher always consumes the payload's own bytes.** After locating, the `jq` path -re-encodes the block and finds that byte sequence in the payload; **the matcher then reads the -payload at that position**, never the string `jq` produced. Two conditions must hold or the -payload is **`unrecognized`**: the re-encoded block must occur **exactly once** in the -payload, and the occurrence must lie **within the located `tool_response` span**. - -Both conditions are load-bearing. "Appears somewhere" is not enough: a canonical copy sitting -in `tool_input` would satisfy it for a response that is *not* canonical, and a non-canonical -escape appearing after an otherwise-canonical marker would leave the two environments -disagreeing about the same bytes. Requiring a unique in-span occurrence, and feeding the -matcher the original bytes, makes both environments read the same input or refuse together — -so the divergence is *removed* rather than documented. - -**Mislocation is safe by construction.** If the block cannot be resolved unambiguously — the -key appears more than once and depth cannot be established, or the canonical-form check above -fails — the payload is **`unrecognized`**, which counts and discloses. Locator uncertainty +this repo's own gate prompts quote payload text. The scan is not a heuristic: it establishes +depth by walking quote state and brace nesting, so a `tool_response` mentioned inside any +string is never a candidate. + +**The matcher consumes the payload's own bytes** — the located span, copied out with nothing +decoding, normalizing or re-encoding it in between. + +**Mislocation is safe by construction, and that is the locator's whole contract.** Every +state the scan cannot resolve — a repeated depth-1 key, a structure it cannot walk, a value +that is not the settled shape — returns *cannot determine* rather than a guess, and the +payload is **`unrecognized`**, which counts and discloses. Locator uncertainty never produces a *wrong verdict*; it produces a counted pass whose disclosure is **attempted and normally persisted**. Where both the emit and the pending write fail (§5.2) that count is silent, so "never silent" would be false: the verdict is never wrong, the disclosure is @@ -132,9 +135,9 @@ reordering degrade to `unrecognized` rather than to a wrong verdict. **Whitespace tolerance** between key, colon and value. Only the current serializer's two-space form has been observed; a formatting change should not silently reclassify. -**Neither path parses the result text as nested JSON.** `jq` decodes it as a string value -and re-encodes it; the scan never decodes at all. What neither does is run a JSON parser -*into* the result string, which is what the open escaped-quote defect makes unsafe. +**The result text is never parsed as nested JSON.** The scan never decodes it at all — it +copies out the escaped bytes and the matcher reads those. What it does not do is run a JSON +parser *into* the result string, which is what the open escaped-quote defect makes unsafe. ### 3.3 The five classes @@ -147,20 +150,18 @@ refers to it rather than restating it. | `failure` | located block's immediate-first property is `success: false` | no | no | | `backgrounded` | the notice anchor (§4), at start of the located block | no | no | | `no-result` | an **unambiguous** determination that no located block yields a non-blank string: `tool_response` absent, `null`, empty array, non-array container, non-object elements, no `text`-type element, `text` not a string, or blank text | no | no | -| `unrecognized` | everything else — a located block matching no anchor, **and every case where locating itself is uncertain**: ambiguous boundary, repeated depth-1 `tool_response` key, or a failed uniqueness/in-span check (§3.1) | bump | store | +| `unrecognized` | everything else — a located block matching no anchor, **and every case where locating itself is uncertain**: a structure the scan cannot walk, or a repeated depth-1 `tool_response` key (§3.1) | bump | store | Stating `no-result` by its complement is deliberate: a hooks-API shape nobody anticipated lands in the **fail-closed** class, which is the direction that matters, since `unrecognized` counts. -**"Blank" is defined on the shared representation, not semantically**, or the two environments -could disagree on a state-changing boundary. The located block is blank when its **escaped -bytes** contain nothing but ASCII space and the two-byte escapes `\n`, `\t`, `\r`. A -Unicode-escaped space (`\u0020`) is *not* blank by this rule, and does not need to be: such a -block fails the canonical-form check first. **Order is fixed** — canonical-form validation -(§3.1) runs *before* the blank test, so every encoding `jq` would normalize has already been -routed to `unrecognized` and never reaches this comparison. §11 item 3 carries the full -encoding table. +**"Blank" is defined on the located bytes, not semantically.** The located block is blank +when its **escaped bytes** contain nothing but ASCII space and the two-byte escapes `\n`, +`\t`, `\r`. A Unicode-escaped space (the six bytes `\u0020`) is *not* blank by this rule, and needs no +special handling to reach the right place: it matches no anchor, so the terminal default +carries it to `unrecognized` — counted and disclosed, never silently read as empty. §11 +item 3 carries the full encoding table. **Malformed outer JSON is not a class at all, and pass 6 got this wrong.** The hook derives `hook_event_name` and `tool_name` from the same document; if it is malformed, the hook cannot @@ -372,8 +373,8 @@ perform it. anchor no longer matches, so the remedy is the same `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` guidance the `backgrounded` message carries; - a mapped third-party tool whose envelope this hook cannot read — **no user-side fix**; - - an ambiguous or non-canonically-encoded payload (§3.1) — this resolves the same way in - both environments, so installing `jq` does **not** change it; + - a payload the locator refuses as ambiguous (§3.1) — installing or removing `jq` does + **not** change this, since classification never consults it; - a hook parser defect — **no operator fix**; the check is to run the hook against the captured payload and report the mismatch. @@ -449,10 +450,10 @@ rather than inherited. each neither count nor store. - **`no-result` shape coverage**: absent field, `null`, empty array, non-array container, non-object elements, array without a `text`-type block, non-string `text`, and empty or - whitespace-only `text`, in both parser environments. + whitespace-only `text`. - **Block selection**: a response whose first element is a non-text block followed by a real - text block must classify from the text block, in both parser environments — the case that - distinguishes "first `text`-type element" from "element `[0]`". + text block must classify from the text block — the case that distinguishes "first + `text`-type element" from "element `[0]`". - **Writer-failure coverage**: with stdout closed or failing, in both parser modes, the hook exits 0, writes no shown-marker, retains any existing pending state, and **creates pending when none existed** (the gate-on failed-emit transition). Marker-write failure alone is not @@ -475,15 +476,15 @@ rather than inherited. review quoting both marker literals (must classify `success`); a failed review quoting `\"success\": true` (must classify `failure`). The failure-direction cases matter most — that is where a mistake produces the false ✓. -- **Extraction parity, asserted on the matcher's input.** Every fixture runs through both - locating paths, and the escaped block handed to the matcher must be **byte-identical** - between them. Asserting only the final class would let two locating bugs cancel out and - report a pass; asserting the input catches a locating divergence where it lives. The - single matcher then needs no parity assertions of its own — there is only one of it. -- The former divergence cases (ambiguous boundary, Unicode-escaped marker, - non-canonical encoding) now assert **`unrecognized` in BOTH environments** — the - canonical-form check makes the `jq` path reach the raw scan's verdict. A test asserting a - precise class with `jq` would pin the very divergence §3.1 removes. +- **Locator-refusal cases**: a repeated depth-1 `tool_response` key, a Unicode-escaped + marker, and an unwalkable structure each assert `unrecognized`. These were the divergence + cases under the two-locator design; with one locator they are ordinary contract cases and + need no cross-environment assertion. +- **`jq`-free mode covers what still branches on `jq`.** Classification does not — there is + one locator — so it needs no parity suite. `field`, `input_field` and `emit` do, and the + suite runs the discarded-class state effects, the writer-failure path and one composed + emit under a `jq`-free `PATH`, because those reach the fallback escaper. Claiming a parity + matrix over classification would test an equivalence the design no longer has. - Golden assertions on both output fields for every message. - The hook exits 0 on every path, including an unreadable result and a marker-write failure. @@ -576,15 +577,14 @@ here because the plan must carry them explicitly at its top — a deferred oblig lives only in a review artifact is one nobody inherits. The plan's own Gate A pins each against real code. -1. **The `jq`-free scanner as a state machine.** Quote state, consecutive-backslash parity, - value boundaries, and what "depth 1" means operationally. §3.1 states the *contract* - (structural with `jq`; conservative and non-guessing without); the plan states the - machine. +1. **The scanner as a state machine.** Quote state, consecutive-backslash parity, value + boundaries, and what "depth 1" means operationally. §3.1 states the *contract* — + structural, conservative, non-guessing; the plan states the machine. 2. **Duplicate and malformed key cases.** Which candidate wins when a depth-1 key repeats, and what happens on malformed JSON. §3.1 sends ambiguity to `unrecognized`; the plan defines what counts as ambiguous. **Which block is selected is NOT deferred** — §3.1 settles it as the first array element whose `type` is `text`, and the plan defines only - how the `jq`-free scanner locates that already-settled block. + how the scanner locates that already-settled block. 3. **Accepted raw encodings for JSON whitespace.** Captured inner newlines are the two bytes `\n`, which POSIX `[[:space:]]` does not match. Every accepted encoding around each prefix token is enumerated in the plan, with compact, tab and CRLF fixtures. From 33ae3e16a9aa34ed4cd463f8678439739369a3c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sun, 2 Aug 2026 15:55:41 +0200 Subject: [PATCH 09/10] feat(hooks): classify gate results and stop counting the ones that reviewed nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validation evidence — docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md battery: full quality command green — shellcheck (6 files), the hook suite 467/467 with 0 failures and 1 named skip under BOTH shells, check-invariants 123/123 + ok, check-version-bump 36/36 + ok against BASE 0dc93d8, claude plugin validate --strict passed. "Both shells" is now literal: HOOK_SH selects the shell the HOOK runs under, and the sh and dash runs produce byte-identical row sets apart from the banner naming the interpreter. An earlier draft of this entry claimed sh-and-dash while every runner invoked the hook via `sh` — bash on macOS — so only ~7 explicit rows ever reached dash. Gate B pass 1 caught it; the runner was parameterized rather than the claim softened. check 1 (counterfactual for the classifier): the 0.8.0 suite and fixtures run against the pre-change hook materialized from BASE in a temp directory. New hook: exit 0, 0 FAIL. Base hook: exit 1, 144 FAIL (measured against the suite as it stood at that point; the suite has grown since with the pass-2 and pass-3 oracles, so the base-hook totals are a record of that run rather than of the current row set). Four named rows, verbatim from the base run and absent from the new one: FAIL - failure/review/default/jq preserves passCount/freshCount/fingerprint/passCountA FAIL - failure/review/default/jq writes no gate-pass state from clean FAIL - backgrounded/review/default/jq writes no gate-pass state from clean FAIL - class: fixture shape2-executor-timeout (got [success], want [failure]) The base hook classifies every captured failure, timeout and backgrounding notice as `success`. The invariant-1 regression is counterfactual too: FAIL - dash: exits 0 with a directory at the marker path (got 2) check 2 (counterfactual for the scan rewrites, each written BEFORE its fix): timed regression rows assert bounds and each failed first. FAIL - perf: 150KB payload classified within 2s (took 11s) -> now 0s FAIL - perf: anchor-prefixed near miss (no-newline) within 2s (took 6s) -> now 1s FAIL - perf: anchor-prefixed near miss (late-newline) within 2s (took 5s) -> now 0s Three quadratics, each measured: `readstr` accumulated byte-by-byte while `substr(s,j,1)` is O(len) per call in BWK awk; the `backgrounded` test ran `${b%%\n*}` on every block, which bash 3.2 evaluates by trying successively longer suffixes; and guarding that on the anchor prefix still left it running in full for a block that starts with the anchor and never completes the notice — the first timed fixture began with an envelope, took the guard's cheap path, and was blind to that branch entirely. check 3 (correctness, found at Gate-B pass 2 and fixed here): with a pending disclosure owed, an event name the hook could not route emitted a MALFORMED document — reproduced as `"hookEventName":"Bogus\\","`, where the trailing backslash escapes the closing quote and Claude Code receives invalid JSON. Two defects introduced by this diff combined: `flush_notes` ran unconditionally so an unroutable event reached `emit`, and `$event` was interpolated raw beside two escaped fields (the jq branch was never affected — `--arg` encodes it). Both fixed. Five oracle rows across both runners, written before the fix and failing first; the jq-free rows are the ones that caught the malformed document, so testing one runner would have missed it. check 5 (correctness, found at Gate-B pass 3 and fixed here): the tool-mapping parser accepted any plausible token, and the mapped cases are tested BEFORE the native `Bash` and `Skill` cases — the only two names outside `mcp__codex__*` that the hooks matcher delivers at all. So `reviewTool=Bash` made a `git commit` COUNT a Gate-B pass instead of resetting the cycle, and `execTool=Skill` counted a skill invocation as a Gate-A pass. Both are false checkmarks in recorded state, reachable from a plausible typo, and this diff is what introduced the contract they contradicted. Reproduced, fixed by requiring mapped names to lie in the namespace, and pinned by three rows including one proving a legitimate in-namespace mapping still counts. check 6 (the 4096-unit notice bound, both sides): the bound that made the anchor-prefixed near miss cheap also NARROWS the backgrounded class — a notice whose segment falls past it counts instead of being discarded. Now tested with a genuine notice immediately inside the cutoff (-> backgrounded) and immediately outside it (-> unrecognized), and stated in spec §4 rather than only in the implementation comment. check 4 (spec conformance): spec §6 fixes the composed order as disclosure first, then the per-occurrence message. The implementation appended instead and the golden froze the inversion. Implementation moved to match the approved spec; goldens updated. WHAT IS NOT BOUNDED, and this entry says so rather than repeating a claim that has now needed narrowing three times. TWO paths, not one: (a) `skipval` walks containers one character at a time, so a large VALID sibling container before `tool_response` costs 3.2 s at 200 KB and 11.5 s at 400 KB; (b) the record accumulator `s = s $0 "\n"` rebuilds the whole input once per input line, so a newline-rich payload is quadratic in line count independently of (a) — 0.35 s at 4k lines, 2.69 s at 16k. Naming only (a) was itself a finding at pass 3: every timed row uses a large single-record payload and never reaches (b). Only the 1 Mi-unit ceiling stops that path, and a payload just under it still costs tens of seconds. Gate-B pass 2 found this after pass 1 found the first two; the earlier drafts of this entry claimed "the curve is now linear", generalizing from the one shape they had measured. That claim is withdrawn: the measured shapes are bounded, the container walk is not, and no timed row covers a branch its own fixture does not reach. verification (named), part 1 — replay of the captured payloads through the changed hook, 9/9 rows, every row guarded on fixture-read status, sed status AND effect, the routed tool name, the exact class and an explicit per-row expected value, with an exact row-count check. The guards were negative-checked in three directions before the green result was accepted: a wrong expected class, a missing fixture, and a wrong row count each fail the step. verification (named), part 2 — story criterion 10 was AMENDED, human-confirmed 2026-08-02, and the amendment is in the story itself with what it gives up stated. The live probe re-run is dropped, not deferred: it requires instrumenting the installed hook, which serves every concurrent Claude Code session here, and no isolated profile can be driven from this session. awk portability — NOT established against this tree at the time of this commit. The local battery runs one awk; CI's ubuntu runner uses mawk, and CI now runs the suite twice (sh and dash) so that run is also the only place the locator meets a second awk implementation. That result binds at PR time and this commit body does not carry it. 467/467 local is shell coverage, not awk-implementation coverage. rollback — verified by inspection, deliberately not executed, per an explicit decision recorded 2026-08-02. The plugin cache IS version-keyed (cache/dev-workflow-kit/dev-workflow//) and installed_plugins.json selects the active one. 0.7.1 is PRESENT and its hook bytes are bee47e59f9b9d682cbfc68a5db0351bf183a27fb, matching the repo bytes at 0dc93d8 — the newest commit whose manifest still contains 0.7.1 — so that resolution method is confirmed here empirically rather than assumed. This repo has NO tags, so "released bytes" remains a repository-bytes claim. NOT verified, and named rather than implied: the operator switch path. 0.8.0 has never been installed, `claude plugin marketplace add` takes no version, and editing installed_plugins.json is undocumented and untested. Rollback is the CURRENT state of this machine, not a procedure anyone has run — which also means every gate-pass counter in this cycle was maintained by 0.7.1, the hook that counts failed and backgrounded calls. The counters are worth nothing here; each pass was judged on its findings file, per CLAUDE.md section 5. What a rollback costs is in the CHANGELOG, not only what it leaves behind: it restores the original defect and the dash special-builtin exit, and it does not undo the three diagnostic markers already in .context/. gate B — closed clean-with-dispositions at pass 9 of 9. Pass 9 returned ZERO new Blocker/Major on both branches; the spec branch was dispositions-only. Four findings are dismissed with a named home in todos.md and a trigger, listed in .context/codex-reviews/gate-b-0.8.0-dispositions.md: the two quadratic locator paths (skipval's container walk and the record accumulator), A5 marker-matrix breadth, A6 composition breadth, and the hardening ledger's missing supersession convention. One one-clause docs precision fix was applied after pass 9 and was NOT re-reviewed; it changed no behaviour, prompt or test. Both pass-9 branches independently reproduced this evidence from their own checkout rather than reading it. What the cycle caught, each fixed with an oracle written before the fix: a denial of service (150 KB of ordinary review text cost 10.9 s in one synchronous hook invocation, now 0.46 s, three separate quadratics); invalid emitted JSON, where an unroutable event plus one unescaped field produced "hookEventName":"Bogus\","; a false-checkmark configuration hijack, where reviewTool=Bash made a git commit COUNT a Gate-B pass instead of resetting the cycle; and a composition order that contradicted spec section 6 while its own golden froze the inversion. Five overclaims in this evidence entry were caught and narrowed across the nine passes. --- .github/workflows/ci.yml | 12 +- AGENTS.md | 5 +- CLAUDE.md | 45 +- README.md | 40 +- docs/architecture.md | 1 + ...6-08-01-gate-pass-result-classification.md | 37 +- .../2026-07-20-codex-file-first-output.md | 19 +- ...iled-codex-call-counts-as-a-pass-design.md | 65 +- ...ailed-codex-call-counts-as-a-pass-story.md | 30 +- .../dev-workflow/.claude-plugin/plugin.json | 2 +- plugins/dev-workflow/CHANGELOG.md | 103 ++ .../dev-workflow/commands/workflow-init.md | 85 +- plugins/dev-workflow/hooks/codex-gate.sh | 522 ++++++- plugins/dev-workflow/hooks/codex-gate.test.sh | 1250 ++++++++++++++++- plugins/dev-workflow/hooks/fixtures/README.md | 143 ++ .../collision-failure-quotes-true.json | 1 + ...ollision-failure-quotes-true.response.json | 1 + .../collision-success-quotes-both.json | 1 + ...ollision-success-quotes-both.response.json | 1 + .../hooks/fixtures/shape0-success-review.json | 1 + .../shape0-success-review.response.json | 1 + .../hooks/fixtures/shape0-success.json | 1 + .../fixtures/shape0-success.response.json | 1 + .../hooks/fixtures/shape1-fast-fail.json | 1 + .../fixtures/shape1-fast-fail.response.json | 1 + .../fixtures/shape2-executor-timeout.json | 1 + .../shape2-executor-timeout.response.json | 1 + .../fixtures/shape3-backgrounding-notice.json | 1 + .../shape3-backgrounding-notice.response.json | 1 + todos.md | 110 +- 30 files changed, 2311 insertions(+), 172 deletions(-) create mode 100644 plugins/dev-workflow/hooks/fixtures/README.md create mode 100644 plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.json create mode 100644 plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.json create mode 100644 plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape0-success-review.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape0-success-review.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape0-success.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape0-success.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.response.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.json create mode 100644 plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.response.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aaaeb86..6c8b3ce 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -70,8 +70,16 @@ jobs: docker run --rm -v "$PWD:/mnt" -w /mnt koalaman/shellcheck:v0.11.0 \ --shell=sh scripts/check-version-bump.test.sh - - name: Hook state-machine tests - run: sh plugins/dev-workflow/hooks/codex-gate.test.sh + # Two runs, because the hook has to be correct under both shells and the runner's + # /bin/sh is dash while a contributor's may be bash. HOOK_SH selects the shell the + # HOOK runs under; without it, running the file under dash would only exercise the + # harness under dash. Ubuntu also gives mawk here, where a developer machine gives + # BWK or GNU awk — this step is the only place the locator meets a second awk. + - name: Hook state-machine tests (sh) + run: HOOK_SH=sh sh plugins/dev-workflow/hooks/codex-gate.test.sh + + - name: Hook state-machine tests (dash) + run: HOOK_SH=dash dash plugins/dev-workflow/hooks/codex-gate.test.sh # Invariants 5 and 6 plus two prompt-conformance checks, mechanically, and BOTH # checkers' regression suites. The diff --git a/AGENTS.md b/AGENTS.md index 4a73177..e0678ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,6 +45,7 @@ plugins/dev-workflow/ agents/finding-triage.md # read-only PR-comment checker (convention-loaded) commands/{workflow-init,process-pr-review}.md hooks/{hooks.json,codex-gate.sh,codex-gate.test.sh} + hooks/fixtures/ # captured PostToolUse payloads the suite replays examples/ # ships, but never scaffolded — one stack's answers docs/ architecture.md # layout + the two non-obvious design decisions @@ -241,10 +242,10 @@ Every command below was run in this session and observed to exit 0. | Role | Command | |---|---| -| quality (the whole battery — what CI runs) | `shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && shellcheck --shell=sh --exclude=SC2015 plugins/dev-workflow/hooks/codex-gate.test.sh && shellcheck --shell=sh scripts/check-invariants.sh && shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && shellcheck --shell=sh scripts/check-version-bump.sh && shellcheck --shell=sh scripts/check-version-bump.test.sh && sh plugins/dev-workflow/hooks/codex-gate.test.sh && sh scripts/check-invariants.test.sh && sh scripts/check-invariants.sh && sh scripts/check-version-bump.test.sh && sh scripts/check-version-bump.sh main && claude plugin validate . --strict` | +| quality (the whole battery — what CI runs) | `shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && shellcheck --shell=sh --exclude=SC2015 plugins/dev-workflow/hooks/codex-gate.test.sh && shellcheck --shell=sh scripts/check-invariants.sh && shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && shellcheck --shell=sh scripts/check-version-bump.sh && shellcheck --shell=sh scripts/check-version-bump.test.sh && HOOK_SH=sh sh plugins/dev-workflow/hooks/codex-gate.test.sh && HOOK_SH=dash dash plugins/dev-workflow/hooks/codex-gate.test.sh && sh scripts/check-invariants.test.sh && sh scripts/check-invariants.sh && sh scripts/check-version-bump.test.sh && sh scripts/check-version-bump.sh main && claude plugin validate . --strict` | | typecheck | n/a — no typed sources (shell + markdown) | | lint | `shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && shellcheck --shell=sh --exclude=SC2015 plugins/dev-workflow/hooks/codex-gate.test.sh && shellcheck --shell=sh scripts/check-invariants.sh && shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && shellcheck --shell=sh scripts/check-version-bump.sh && shellcheck --shell=sh scripts/check-version-bump.test.sh` | -| test | `sh plugins/dev-workflow/hooks/codex-gate.test.sh` | +| test | `HOOK_SH=sh sh plugins/dev-workflow/hooks/codex-gate.test.sh && HOOK_SH=dash dash plugins/dev-workflow/hooks/codex-gate.test.sh` — two runs; `HOOK_SH` selects the shell the HOOK runs under, and without it a dash invocation only exercises the harness | | invariant checks (5 pinning, 6 manifest, prompt conformance) | `sh scripts/check-invariants.test.sh && sh scripts/check-invariants.sh` | | invariant check (12 version bump) | `sh scripts/check-version-bump.test.sh && sh scripts/check-version-bump.sh main` | | build | n/a — nothing is compiled or bundled | diff --git a/CLAUDE.md b/CLAUDE.md index a2675b9..0363e84 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,17 +165,27 @@ write or just returns its prior summary is not established; if it returns the su that was the attempt. Spent and still incomplete → STOP and surface, naming which check failed. -**What this does not do.** The hook counts on `PostToolUse`, keyed on tool name, and -never sees the file. Claude Code fires `PostToolUse` after a *successful* call and routes -a failed one to `PostToolUseFailure`, which the plugin registers no handler for — but do -not infer from that which failures escape counting: the pinned `mcp-codex-dev` catches -its own errors, executor timeouts and aborts included, and returns them as a normal -result carrying `success: false` rather than throwing or setting `isError` -(`dist/tools/codex-review.js`). A failed review therefore looks like a successful tool -call and increments the counter. So does a call that returns and then fails validation. -The rule that follows is the simple one: **discount every incomplete pass regardless of -what the counter says** — a "satisfied" count can overstate the passes you actually hold, -and reasoning about which failure took which event path will get it wrong. Nothing checks the terminator mechanically; this is +**What this does not do.** The hook counts on `PostToolUse`, keyed on tool name **and on +the result envelope**, and still never sees the file. Claude Code fires `PostToolUse` +after a *successful* call and routes a failed one to `PostToolUseFailure`, which the +plugin registers no handler for — but do not infer from that which failures escape +counting: the pinned `mcp-codex-dev` catches its own errors, executor timeouts and aborts +included, and returns them as a normal result carrying `success: false` rather than +throwing or setting `isError` (`dist/tools/codex-review.js`). A failed review therefore +still looks like a successful *tool call* — but as of 0.8.0 the hook reads the result of +gate calls it can route, and withholds the count for three **recognized** shapes: an +envelope whose **first** property is `success: false`, the harness backgrounding notice +**in the wording it currently uses**, and a result from which no usable text can be +obtained. Every other routed gate call counts, including any located text the hook cannot +interpret — a reordered envelope, a reworded notice, an unknown third-party shape — which +counts **with** a disclosure that is attempted and normally shown once per workspace, but +can be lost or repeated when its marker cannot be persisted. So does a call that returns +and then fails validation. The counter is therefore closer to the truth than it was and +is still not evidence: a "satisfied" count can still overstate the passes you actually +hold, and reasoning about which failure took which event path will get it wrong. The rule +that follows is the simple one: **discount every incomplete pass regardless of what the +counter says**, because classification cannot see whether the findings file was written. +Nothing checks the terminator mechanically; this is instruction-backed by design, and a recurring truncation incident is the trigger to build the checker, not a reason to build it now. Detection is conditional: it catches an absent or malformed terminator, a count mismatch and a missing branch file *in the artifact you @@ -395,11 +405,14 @@ like the rest of §5; the detection is a reader comparing the pass against the s - **Timeout / abort:** a codex call that dies at the MCP tool-call timeout is retried once before surfacing to the user, and that retry *is* the single shared recovery attempt above — not a second one. An abort is an incomplete pass, so treat it as one: - it may already have moved the hook's counter (the pinned server returns its own - timeouts as ordinary results), and it may have left a partial or stale target file, so - delete the targets and confirm them gone before retrying, then validate the result like - any other pass. Counter and workspace state persist in `.context/`; the *pass* does - not. + it may have left a partial or stale target file, so delete the targets and confirm them + gone before retrying, then validate the result like any other pass. Whether it moved the + hook's counter depends on the shape it returned and on which hook version is installed: + as of 0.8.0 a recognized failure envelope, the recognized backgrounding notice and a + result yielding no usable text are all withheld from the count, while a reordered, + reworded or unrecognized shape still counts fail-open. Do not reason from the counter + either way — an incomplete pass is discounted whatever it says. Counter and workspace + state persist in `.context/`; the *pass* does not. ## 6. Context Canary diff --git a/README.md b/README.md index e721142..310205d 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,10 @@ old version until you restart it or run `/reload-plugins`. - **Codex** — the reviewer behind both gates. Needs the **Codex CLI, authenticated with an OpenAI account** — a real external dependency, not just the `.mcp.json` entry `/workflow-init` writes for you. It also needs **a Codex MCP server that exposes - `exec` and `review`** — the gates and their pass counters key on those two tool names. + `exec` and `review`** — the gates key on those two tool names, and the pass counters + additionally skip routed calls whose result the hook reads as failed, backgrounded, or + yielding no usable text; a result it can read but not interpret still counts, and + normally says so once. Use the `mcp-codex-dev` server `/workflow-init` pins, which has both. The official `codex mcp-server` is a *different* server exposing a single `codex` tool, which can't be attributed to Gate A (reviews text) or Gate B (reviews a diff): with it connected, @@ -73,6 +76,37 @@ old version until you restart it or run `/reload-plugins`. findings out of the response. - **`gh`** — optional; only `/dev-workflow:process-pr-review` uses it. +**2b. Set `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`** — strongly recommended, and the primary +defence against a gate call being auto-backgrounded. Claude Code moves a long MCP call to +the background at a threshold (120 s by default) and fires `PostToolUse` *at that moment*, +carrying the harness's own notice instead of any Codex result; the eventual real +completion fires no second `PostToolUse`, so the hook never sees the review's outcome. + +Requires **Claude Code 2.1.212 or newer**. Set it in the environment Claude Code is +**launched from** — it is read at process start, so exporting it inside a tool shell +leaves the running session unchanged and you must restart Claude Code. Use `0` to disable +auto-backgrounding, or a positive value that **exceeds** your longest expected gate call; +a positive value shorter than the call still backgrounds it. + +```sh +export CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS=0 # or e.g. 900000, exceeding your longest gate call +``` + +**Without it, both outcomes are possible, and only one is safe.** While the harness notice +keeps the wording the hook recognizes, the backgrounded pass is *discarded* — correct, but +**re-running it is not simply a retry**: the original call may still be running and can still +write into the findings slot afterwards, leaving a correctly terminated file from the wrong +run that no downstream check can detect. Stop that task by the id in the tool result — or +await it if the result carries none — and delete the findings slot first. The hook's own +backgrounding message carries that procedure; follow it there rather than from memory. If that harness prose ever changes, the hook can no longer recognize +the notice and the call is *counted* instead, with a disclosure that says the count was +made without inspection. That second outcome is why this setting is the defence rather +than the hook: the hook fails safe against today's wording, not against every future one. + +(`CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS` appears in the 2.1.220 string table but was +never exercised here, so it is deliberately not documented — naming the wrong one of two +similar variables is this project's own docs-drift class.) + **3. Run `/dev-workflow:workflow-init` in each project.** It verifies the rest and tells you what's missing — git repo, superpowers, Codex (not configured / not loaded / ok), `gh`, `AGENTS.md`, stack — before writing a single file. Then follow @@ -94,8 +128,8 @@ plugin is installed once per machine; every other repo you open hears nothing fr Per-workspace knobs, all files under `.context/`: | `codex-gate.floor` | a positive integer; moves the 3-passes-per-gate floor. | -| `codex-gate.off` | silences the reminders; state keeps tracking, so re-enabling is accurate. | -| `codex-gate.tools` | `execTool=` and/or `reviewTool=` — counts a Codex server whose tools aren't named `exec`/`review`, and only worth it if that server really does separate text-review from diff-review; aiming both gates at one general-purpose tool moves the counters while neither gate means what it says. Unparseable lines are ignored, so a typo can't quietly unhook a gate. | +| `codex-gate.off` | silences the reminders; classification and state tracking keep running, so re-enabling lands on counters carrying the same semantics as gate-on — which is not the same as evidence that a review happened. | +| `codex-gate.tools` | `execTool=` and/or `reviewTool=` — counts a Codex server whose tools aren't named `exec`/`review`, and only worth it if that server really does separate text-review from diff-review; aiming both gates at one general-purpose tool moves the counters while neither gate means what it says. Each mapped name must itself lie in `mcp__codex__*`: the hook's `hooks.json` matcher is `^(Bash\|Skill\|mcp__codex__.*)$`, so an out-of-namespace name is either never delivered (the mapping looks applied and does nothing) or, for the reserved names `Bash`/`Skill`, hijacks a lifecycle event; the hook refuses both — register the server as `codex` to place its tools there. Unparseable lines are ignored, so a typo can't quietly unhook a gate. | **Without Codex**, `/workflow-init` degrades honestly instead of scaffolding gates that can't run: it silences the hook and marks CLAUDE.md §5 `INACTIVE` with the re-enable diff --git a/docs/architecture.md b/docs/architecture.md index 4e9a8c5..d896fbb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -24,6 +24,7 @@ plugins/dev-workflow/ agents/finding-triage.md commands/{workflow-init,process-pr-review}.md hooks/{hooks.json,codex-gate.sh,codex-gate.test.sh} + hooks/fixtures/ # captured PostToolUse payloads the suite replays examples/ # ships, but never scaffolded — one stack's answers docs/{getting-started,coding-workflow,prompt-standards,architecture}.md docs/{hardening-log,hardening-taxonomy,pr-review-bots}.md diff --git a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md index ee4397d..ca96ec2 100644 --- a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md +++ b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md @@ -94,7 +94,8 @@ shellcheck --shell=sh scripts/check-invariants.sh && \ shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && \ shellcheck --shell=sh scripts/check-version-bump.sh && \ shellcheck --shell=sh scripts/check-version-bump.test.sh && \ -sh plugins/dev-workflow/hooks/codex-gate.test.sh && \ +HOOK_SH=sh sh plugins/dev-workflow/hooks/codex-gate.test.sh && \ +HOOK_SH=dash dash plugins/dev-workflow/hooks/codex-gate.test.sh && \ sh scripts/check-invariants.test.sh && \ sh scripts/check-invariants.sh && \ sh scripts/check-version-bump.test.sh && \ @@ -102,7 +103,7 @@ claude plugin validate . --strict ``` Every later task means **this**, run and green — re-read it against the `AGENTS.md` row before the first commit, since that row is the source of truth and this is a copy. -- **Plus a `dash` run of the hook suite** — `dash plugins/dev-workflow/hooks/codex-gate.test.sh` — from Task 4 onward, where the marker writes begin. The special-builtin defect above is shell-dependent and macOS `sh` does not expose it; claiming `dash` coverage without running `dash` is what let it ship. If `dash` is unavailable, say so and record CI-on-Ubuntu as the only evidence, rather than claiming both. +- **The `dash` run needs `HOOK_SH=dash`, and without it the run is theatre.** Every runner in the suite invokes the HOOK through `$HOOK_SH_BIN`; the variable defaults to `sh`, so `dash codex-gate.test.sh` alone executes the harness under dash and the hook under whatever `/bin/sh` is — bash, on macOS. That is exactly the gap Gate-B pass 1 found in this cycle's own release evidence, after the earlier wording here invited it. The special-builtin defect is shell-dependent and macOS `sh` does not expose it; claiming `dash` coverage without running the HOOK under dash is what let it ship. If `dash` is unavailable, say so and record CI-on-Ubuntu as the only evidence, rather than claiming both. - **After committing** — `sh scripts/check-version-bump.sh "$BASE"`, with `$BASE` the SHA recorded at Task 1. It compares commits, so the commit must exist first; and passing `main` on this checkout compares HEAD with itself and succeeds trivially, which is not evidence. ## File Structure @@ -293,13 +294,15 @@ Commit: `WIP: refactor(hooks): buffer output so one invocation writes one docume **One task and one commit, deliberately.** Splitting locator, classifier, wiring and disclosure produced intermediate commits that could not be green: the classifier is unobservable until it is wired, and the wiring calls the diagnostic interface. Each such commit would ship either a function nothing calls or a call to a missing function. -**A1 — the state machine, stated operationally.** The scan holds the whole payload in one string and walks it with three primitives. `readstr` consumes a JSON string from its opening quote and returns the index of its closing quote, accumulating the **raw** bytes: **backslash parity falls out of an `esc` flag** — a backslash takes the next byte verbatim whatever it is, so `\\` ends parity and the following `"` closes the string, while `\"` does not. `skipws` consumes space, tab, newline and CR. `skipval` consumes one **span**: a quote-aware string; a balanced container span whose closers must match their openers (`[1}` is rejected); or a primitive, only if the token is exactly `true`, `false`, `null` or a JSON number. It does **not** parse a container's members. **"Depth 1" is not a counter** — it is the structural position the walk occupies: the top-level loop reads key/colon/value triples of the payload's single outer object and nothing else, so a `tool_response` inside any string or nested container is never a candidate. +**A1 — the state machine, stated operationally.** The scan holds the whole payload in one string and walks it with three primitives. `readstr` consumes a JSON string from its opening quote and returns the index of its closing quote, yielding the **raw** bytes between them. **Backslash parity is the whole rule** — a backslash takes the next byte verbatim whatever it is, so `\\` ends parity and the following `"` closes the string, while `\"` does not. It was written as a byte-at-a-time `esc`-flag loop; that loop was quadratic (`substr(s,j,1)` is O(len) per call in BWK `awk`), so it is now **one anchored `match()`** encoding the same parity rule, extracting once with `substr`. Same accepted language, same bytes returned — the mechanism changed, not the contract. `skipws` consumes space, tab, newline and CR. `skipval` consumes one **span**: a quote-aware string; a balanced container span whose closers must match their openers (`[1}` is rejected); or a primitive, only if the token is exactly `true`, `false`, `null` or a JSON number. It does **not** parse a container's members. **"Depth 1" is not a counter** — it is the structural position the walk occupies: the top-level loop reads key/colon/value triples of the payload's single outer object and nothing else, so a `tool_response` inside any string or nested container is never a candidate. **A2 — recognition, and what carries the safety. `locate_result` is a locator, not a validator**, and this paragraph claims nothing more. Three passes read a stronger promise into the prose than the code keeps; the promise is stated at its true size and pinned by fixtures rather than by wording. **What makes the walk trustworthy is string-boundary tracking, and nothing else.** `readstr` decides where every JSON string starts and ends, from quote state and backslash parity alone. That one property is why a `tool_response` mentioned *inside* a string — the `tool_input` decoy, a result quoting the key, this repo's own gate prompts — is never mistaken for the key. **Malformation outside a string boundary cannot redirect the walk**: it may be stepped over, but it cannot move where the next string begins. -**Refusals it makes**, each tested: a second depth-1 `tool_response` key; a repeated `type` or `text` member in **any element the scan examines**, not only the one it would select (verified: a duplicate inside a non-text element *preceding* a valid text block also refuses — stricter than the first-text-element rule requires, and it is the behaviour, so the prose states it); a document not starting with `{`; a non-string key, missing colon or unterminated string in the path it walks; a mismatched container delimiter; a stray comma; a bare token that is not a JSON literal; garbage after the outer object's `}`; and **nesting deeper than 200**. +**Refusals it makes**, each tested: a second depth-1 `tool_response` key; a repeated `type` or `text` member in **any element the scan examines**, not only the one it would select (verified: a duplicate inside a non-text element *preceding* a valid text block also refuses — stricter than the first-text-element rule requires, and it is the behaviour, so the prose states it); a document not starting with `{`; a non-string key, missing colon or unterminated string in the path it walks; a mismatched container delimiter; **a stray comma in the path it walks**; a bare token that is not a JSON literal; garbage after the outer object's `}`; and **nesting deeper than 200**. + +**The comma rule is scoped to the walked prefix, and the scope is the whole of it.** The scan **exits as soon as it selects a text block**, so nothing after that element is examined at all — a *trailing comma after the selected element* is not refused, it is never seen. The frozen verifier pins that case as `success`, and the suite asserts it. Stating the rule unqualified described a validator the code is not (P9-4). **Refusals it does NOT make, frozen as fixtures asserting today's behaviour.** A *walkable* invalid document is walked past and the real block located, status 0 — `[1,]`, `{"a" 1}` and `"\q"` in a sibling value are pinned so this correspondence is checked mechanically instead of argued again. The response array is walked only as far as the selected element. An escaped key spelling is compared as raw bytes and simply does not match. @@ -307,11 +310,21 @@ Commit: `WIP: refactor(hooks): buffer output so one invocation writes one docume **Two bounds, and what each does and does not cover.** The **length ceiling** refuses a payload whose accumulated length would exceed 1 Mi *`awk` length units* — not bytes: POSIX `awk`'s `length()` counts characters and implementations differ on multibyte input, so the cut-off is not identical between BWK `awk` and `mawk`. It bounds the **scan**, not memory: `payload=$(cat)` already holds the whole input and `awk` still ingests it. The **depth cap** refuses nesting past 200, because the closer stack is a string and each push is O(length), making deep nesting quadratic in work an external MCP result could dictate. Neither is a contract; both are backstops, and both route to `unrecognized`. +**Both bounds are SIZE bounds, and a size bound only bounds work if the work is linear in size — which it was not.** Gate B on the implementation measured a 150 KB text block, comfortably under the ceiling, at **10.9 s in one synchronous hook invocation**, and the cost grew quadratically: ×3 size cost ×8.4 time. Two independent quadratics, both fixed here, both now covered by a timed regression row with a stated bound rather than by prose: +- `readstr` accumulated the value byte-by-byte **and** re-derived `length(s)` per iteration; worse, `substr(s,j,1)` is O(len) per call in BWK `awk`, so any per-character walk over the payload is quadratic on its own. It now finds the string with one anchored `match()` and extracts once. +- The `backgrounded` test evaluated `${b%%\n*}` on every block. `%%` removes the **longest** matching suffix, which bash 3.2 — macOS `/bin/sh`, what the hook runs under — evaluates by trying successively longer suffixes; with no `\n` present that is quadratic, and it dominated everything else (5.4 s at 150 KB, 21.6 s at 300 KB). It is guarded by the cheap literal prefix the anchor already requires **and** truncated to a bounded head before it runs — the guard alone was not enough, as the paragraph below records. + +A third quadratic sat behind the second and was found by the next Gate-B pass: guarding the expansion on the anchor prefix left it running in full for any block that *does* start with the anchor and then never completes the notice — 5.9 s at 150 KB — and the first timed fixture began with an envelope, so it took the guard's cheap path and never reached the branch. The head is now truncated to a bounded 4096 characters before the expansion runs, which costs one documented limit: a notice whose quoted tool name exceeds ~4070 characters is `unrecognized` rather than `backgrounded`, so it counts. + +**What is bounded, and what is NOT — stated this way because three consecutive attempts at this paragraph each claimed more than held.** Bounded now: the ordinary success/failure envelope path (150 KB in 0.46 s), and the anchor-prefixed near miss in both its `\n` placements. **Still quadratic — TWO paths, not one.** (a) `skipval` walks a container character by character, and `substr(s,i,1)` is O(len) per call in BWK `awk`, so a large *valid sibling container before* `tool_response` costs 3.2 s at 200 KB and 11.5 s at 400 KB. (b) The record accumulator `s = s $0 "\n"` rebuilds the whole input once per input line, so a **newline-rich** payload is quadratic in line count independently of (a) — a pretty-printed payload costs 0.35 s at 4k lines, 0.75 s at 8k and 2.69 s at 16k. Naming only (a) was itself a finding at pass 3: the timed rows all use large single-record payloads and never reach (b). Only the 1 Mi-unit ceiling stops that path, which means the ceiling is doing real work rather than serving as a formality — and a payload just under it still costs tens of seconds. So the ceiling does **not** bound the work at roughly two seconds in general; it bounds it at roughly two seconds *on the shapes measured above*. + +**The general claim to keep, because it is the one that was missing:** a size backstop says nothing about time unless the per-byte cost is constant, and in POSIX shell and `awk` it frequently is not — `substr(s,i,1)` and `${var%%pat*}` are both O(len) per operation here. **And the procedural lesson, earned three times in one cycle:** a timed row only covers the branch its own fixture reaches, so "the curve is linear" generalized from one measured shape has been wrong every time it was written. + Malformed JSON is **not a class**: per spec §3.3 a payload the hook cannot *route* never reaches here. **Routability itself depends on `jq`, and that predates this change**: with `jq`, `field()` on a malformed document returns empty and the hook exits silently; without it, the `grep` fallback can still read a tool name, so the same payload routes and lands in `unrecognized`. That divergence belongs to `field()` and is left alone — stated because a reader comparing the two environments will otherwise read it as a classifier bug. -**A3 — the accepted encodings, complete.** Whitespace is accepted at exactly **three** points — after the encoded `{`, after the key `\"success\"`, after the `:` — and at each: literal ASCII space, and the two-byte escapes `\n`, `\t`, `\r`. Nothing else; a Unicode escape is not whitespace here and needs no special handling, because it matches no anchor and the terminal default carries it to `unrecognized`. **Blank** is the same alphabet over the whole block. **The value token must end** at a delimiter — `true` followed by anything other than encoded whitespace, `,` or `}` is `unrecognized`, because `true*` alone accepts `truely`, a false **success**. Whitespace stripping is bounded at 64 units; past it the prefix match fails and the block is `unrecognized`. The blank test is one `sed` pass, so a large blank block costs linear work in `sed` rather than shell iterations. +**A3 — the accepted encodings, complete.** Whitespace is accepted at exactly **four** points — after the encoded `{`, after the key `\"success\"`, after the `:`, and **after the value token**, where `_token_ends` strips before testing for a delimiter — and at each: literal ASCII space, and the two-byte escapes `\n`, `\t`, `\r`. Nothing else; a Unicode escape is not whitespace here and needs no special handling, because it matches no anchor and the terminal default carries it to `unrecognized`. **Blank** is the same alphabet over the whole block. **The value token must end** at a delimiter — `true` followed by anything other than encoded whitespace, `,` or `}` is `unrecognized`, because `true*` alone accepts `truely`, a false **success**. Whitespace stripping is bounded at 64 units; past it the prefix match fails and the block is `unrecognized`. The blank test is one `sed` pass, so a large blank block costs linear work in `sed` rather than shell iterations. -**A4 — the notice grammar, and what is deliberately outside it.** The anchor is exactly: the block **begins** `MCP tool \"` (escaped quote — the shared representation carries `\"`, and a draft expecting a literal `"` did not match the real capture at all), and the segment `\" is still running after ` occurs **before any `\n` escape**. Everything else is variable **by decision**: spec §4 places the quoted tool name, the threshold digits and unit, and the task id outside the anchor so it covers both gate tools, any mapped name and any threshold. Validating a duration format would narrow the anchor to shapes observed once, widening C1 rather than closing it. **Near-misses that must not match**, each tested: the phrase later in the text; a whole notice quoted inside a real envelope's `summary`; a block beginning `MCP tool \"` with no `is still running after`; the segment appearing only after a `\n`. +**A4 — the notice grammar, and what is deliberately outside it.** The anchor is exactly: the block **begins** `MCP tool \"` (escaped quote — the shared representation carries `\"`, and a draft expecting a literal `"` did not match the real capture at all), and the segment `\" is still running after ` occurs **before any `\n` escape**, **within the first 4096 units of the block** — a bound added for performance that narrows this class: a notice whose segment falls past it is `unrecognized` and therefore COUNTS rather than being discarded. ~200x the longest real tool name, tested immediately inside and outside the cutoff, and stated in spec §4. Everything else is variable **by decision**: spec §4 places the quoted tool name, the threshold digits and unit, and the task id outside the anchor so it covers both gate tools, any mapped name and any threshold. Validating a duration format would narrow the anchor to shapes observed once, widening C1 rather than closing it. **Near-misses that must not match**, each tested: the phrase later in the text; a whole notice quoted inside a real envelope's `summary`; a block beginning `MCP tool \"` with no `is still running after`; the segment appearing only after a `\n`. **A5 — the complete marker table.** Three files: `codex-gate.bgAdvice`, `codex-gate.unverified` (shown), `codex-gate.unverifiedPending` (owed). `bgAdvice` is independent of the other two. @@ -321,11 +334,11 @@ Malformed JSON is **not a class**: per spec §3.3 a payload the hook cannot *rou | absent | absent | `unrecognized`, flush suppressed (1) | write `pending` | | absent | absent | `unrecognized`, flush failed (2) | write `pending` | | absent | absent | `unrecognized`, flush wrote, **`unverified` write fails** | write `pending` — the debt survives the marker | -| absent | present | any unsuppressed event | prepend `Earlier: ` disclosure; on 0 → write `unverified`, delete `pending`; on 1 or 2 → **retain** `pending` | +| absent | present | any unsuppressed **routed** event | prepend `Earlier: ` disclosure; on 0 → write `unverified`, delete `pending`; on 1 or 2 → **retain** `pending` | | absent | present | the current event is **itself `unrecognized`** | one disclosure, **no `Earlier:` prefix**, pending cleared on a successful write | | absent | present | flush wrote, `unverified` write fails | **retain** `pending` (duplicate beats loss) | | absent | present | flush wrote, `unverified` written, **`pending` delete fails** | present+present → next row | -| present | present | any event | delete `pending` best-effort at flush start; no disclosure | +| present | present | any routed event | delete `pending` best-effort at flush start; no disclosure | | present | absent | `unrecognized` again | nothing emitted, nothing written | | absent | absent | `unrecognized`, flush **suppressed** and `pending` write fails | counted, nothing delivered, nothing recorded — **C2, direction 1** | | absent | absent | `unrecognized`, flush **failed** and `pending` write fails | counted, nothing delivered, nothing recorded — **C2, direction 2** | @@ -519,11 +532,11 @@ Exclude generated state explicitly: `.mcp/` holds cache copies of the hook (`AGE The unknown-tool message is a shipped prompt, so its replacement is literal. Add the target-model prefix the other four hook prompts carry, then after *"…genuinely has two tools that separate reviewing TEXT from reviewing a DIFF"*, insert: -> **A mapped name must also lie in the `mcp__codex__*` namespace.** This hook is invoked by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming a tool outside it never fires and the gate stays silent — the mapping looks applied and does nothing. +> **A mapped name must also lie in the `mcp__codex__*` namespace.** This hook is invoked by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming a tool outside it is either never delivered — the mapping looks applied and does nothing — or, for the reserved names `Bash` and `Skill` that matcher does deliver, hijacks that lifecycle event. The parser refuses both. (Normative statement: the design's decision 1.) And to its `systemMessage`, because registering a server is an operator action: -> A mapped tool name must start with `mcp__codex__`, or the gate never fires. Register the server under the name `codex` to place its tools there. +> A mapped tool name must start with `mcp__codex__`, or it is refused: outside that namespace it is either never delivered to the hook or — for `Bash` and `Skill` — hijacks a lifecycle event. Register the server under the name `codex` to place its tools there. - [ ] **Step 5: The layout trees.** Add `hooks/fixtures/` to `AGENTS.md` § Architecture and `docs/architecture.md`. Two greps, because the Don'ts require both: `grep -rn 'codex-gate' AGENTS.md docs/architecture.md MANIFEST.md README.md`, and the mandated manifest-claim census `grep -rniE 'declare[sd]?|convention[- ]load' --include='*.md' . | grep -v source-files/`. **Then read `plugins/dev-workflow/.claude-plugin/plugin.json` in this same change** and check every hit against it — three sites once shipped the claim that the manifest declares `hooks` when it declares nothing, and no mechanical check can tell whether a sentence about a manifest is true. @@ -579,6 +592,8 @@ EVIDENCE="$(git rev-parse --show-toplevel)/.context/evidence-0.8.0.md" # .cont **Side-by-side bytes are not an operable rollback.** Record and verify how an operator actually *activates* 0.7.1 in this environment — the cache path is an implementation detail and `claude plugin marketplace add` has no version syntax — and confirm the active hook's bytes afterwards. If no verified switch path exists, stop the release rather than shipping a rollback story nobody has run. + > **AMENDED 2026-08-02, human-confirmed by Daniel. The hard stop above is lifted for this release only, and replaced by a documentation requirement.** The drill is *not executed*; rollback is verified by inspection and its unverified part named. What was established: the plugin cache **is** version-keyed (`cache////`), `installed_plugins.json` is what selects the active version, 0.7.1 is present, and its hook bytes match the repo at the newest commit whose manifest still carries 0.7.1 — so that resolution method is confirmed empirically rather than assumed. What is **not** established, and is stated in the release evidence rather than implied away: no operator switch path has been executed, because 0.8.0 has never been installed and editing `installed_plugins.json` is undocumented. **Two facts make executing it the wrong trade here:** the drill would mutate a shared plugin environment that every concurrent Claude Code session on this machine runs against (the risk P9-24 named), and 0.7.1 **is already the active version**, so the rollback target is the current state rather than a state anyone must reach. The residual is that recovery *after* activating 0.8.0 is untested. Reinstate the hard stop for any later release that ships with 0.8.0 already active, where that residual is no longer hypothetical. + **Then put the candidate back, and prove it.** This drill ends with 0.7.1 *active*. Running it before Gate B and not reversing it means every Gate-B pass executes under the old hook — which counts failed calls, so the pass accounting the review depends on is the accounting this change exists to fix. Either run the whole drill in an isolated profile, or reactivate 0.8.0, **byte-verify the active hook**, and reload the session before the first Gate-B call. Verify, do not assume: this is the one step whose failure is invisible until the review is already worthless. **State what a rollback costs, not only what it leaves behind.** Reverting to 0.7.1 **restores the original defect**: failed, timed-out and backgrounded calls count as gate passes again. It also restores the `dash` special-builtin exit — on Linux, an unwritable `.context/` makes 0.7.1's hook exit 2, violating invariant 1. So rollback is right for "0.8.0 discards passes it should count", and wrong for anything else; a CHANGELOG that mentions only the leftover markers lets an operator roll back believing it merely removes classification. Also record what it does **not** undo: the three diagnostic markers stay in `.context/`, and 0.7.1 ignores them. @@ -648,6 +663,6 @@ Inherited, not dropped — the way this plan inherited spec §11's deferred cont **Spec coverage:** §3.1 → Task 5 (A1, A2); §3.2–3.3 → Task 5; §4 → Task 5 (A4) and C1; §5.1 → Task 5 Step 3; §5.2 → Task 5 Step 4 (A5); §6 → Tasks 4 and 5; §7.1 → Task 1; §7.2 → Task 2; §7.3 → Task 5's oracle tables; §7.4 → Task 7; §9 → Task 6; §11 → A1–A7 as mapped. -**Placeholders:** the harness shell, deliberately and by scope (see the note at the top and the Gate-B list). Not placeholders: the five message pairs, the class and marker tables, the contracts, every oracle, and Task 7's procedures — those are what Gate A can judge. Task 6's prose *targets* are located by line and grep rather than quoted, because quoting them would create a fourth copy of the sentences this change exists to correct. +**Placeholders:** the harness shell, deliberately and by scope (see the note at the top and the Gate-B list). Not placeholders: the five message pairs, the class and marker tables, the contracts, every oracle, and Task 7's procedures — those are what Gate A can judge. **The five pairs are no longer verbatim-final, and this document is not their source of truth:** Gate B revised all five `additionalContext` literals and `UNVERIFIED_MSG` — the target-model prefix (P9-28), the background report format (P9-29), the cleanup rationale and its later deletion (P9-30), the `FAILURE_CTX` scope (P9-31), the refusal families (P9-32), the trust boundary (P9-33), the task-id conditional (P9-36) and the repeat causes (P9-37). The shipped strings in `codex-gate.sh`, and their literal goldens in the suite, are authoritative. Task 6's prose *targets* are located by line and grep rather than quoted, because quoting them would create a fourth copy of the sentences this change exists to correct. **What no reader should infer.** The scan is a locator, not a validator: string-boundary tracking is what makes the walk trustworthy, a walkable-invalid document is walked past (three fixtures pin that), and its two bounds are backstops in `awk` length units and nesting depth, not contracts — neither bounds memory. The marker table describes sequential behaviour only (C4). The `+check` counterfactual shows four named tests depend on the new code, not that the old hook was wrong in every way the new one is right. 56/56 local is shell coverage, not `awk`-implementation coverage. And the oracles state what each test must *fail on*; whether the shell satisfying them is correct is a Gate-B question, which is the point of moving it there. diff --git a/docs/superpowers/specs/2026-07-20-codex-file-first-output.md b/docs/superpowers/specs/2026-07-20-codex-file-first-output.md index 238c789..1984a7b 100644 --- a/docs/superpowers/specs/2026-07-20-codex-file-first-output.md +++ b/docs/superpowers/specs/2026-07-20-codex-file-first-output.md @@ -183,7 +183,24 @@ checks 1–4 failed. ### What this does NOT do (stated so nobody mistakes it for a guard) - **The hook still counts an incomplete pass — specifically the incomplete passes this - protocol is about.** Counters increment in the hook's `PostToolUse` branch + protocol is about.** + + > **SUPERSEDED IN PART as of plugin 0.8.0.** The paragraph below describes the hook up + > to 0.7.1, and its *conclusion* still holds — discount every incomplete pass whatever + > the counter says — but its *mechanism* no longer does. As of 0.8.0 the hook reads the + > result of gate calls it can route and withholds the count for three recognized shapes: + > an envelope whose immediately-first property is `success: false`, the harness + > backgrounding notice in its current wording, and a result yielding no usable text. + > Everything else it routes still counts, including located text it cannot interpret. + > **The normative definition is the five-class table in + > `docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md` §3.3** + > — read that rather than this passage for current behaviour. What survives unchanged is + > the reason this bullet exists: classification cannot see whether the findings file was + > written, so a counted pass is still not evidence of a review. The analysis below is + > kept rather than rewritten because it records how the 0.5.1 belief was corrected, and + > that reasoning is the thing worth not losing. + + Counters increment in the hook's `PostToolUse` branch (`codex-gate.sh:361`), keyed on tool name; the hook never sees the response, let alone the file. Claude Code fires `PostToolUse` after a tool call *succeeds* and routes a failed call to `PostToolUseFailure`, for which `hooks/hooks.json` registers no handler. diff --git a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md index 3f98fbb..54328f0 100644 --- a/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md +++ b/docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md @@ -25,7 +25,7 @@ them. 1. **Classification keys on the result envelope, not the tool name.** A tool mapped via `.context/codex-gate.tools` is in scope — **within the reachable namespace**. The hook is invoked by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming - a tool outside `mcp__codex__*` can never fire, and always could not. The matcher is left + a tool outside `mcp__codex__*` is either never delivered (except the reserved names `Bash` and `Skill`, which the matcher does deliver — so such a mapping hijacks a lifecycle event instead of doing nothing; the parser refuses both). **This paragraph is the single normative statement of the namespace boundary.** The **plan** points here in its own prose; where its Task 6 quotes the literal text to be written into a shipped surface, that quotation restates by construction and carries both halves like the surface it authors. **Everything that ships to another repo restates it deliberately and keeps the copies in sync** — `README.md`, `CHANGELOG.md`, the hook prompts, and `/workflow-init`'s inline `CLAUDE.md` template. That is not laxity: **a scaffolded template cannot point at this file at all**, because the project it is written into does not have it — which is exactly the self-contained exception invariant 8 already states. An earlier revision of this sentence listed the template among the pointers, which invariant 8 makes impossible. A restatement is correct only if it carries **both** halves — never delivered, *or* hijacking `Bash`/`Skill`. The matcher is left as it is: broadening it would start a hook process on every MCP tool call in every adopted workspace, which is a real cost for a case with a one-line remedy — register the third-party server under the server name `codex`, which places its tools in the reachable @@ -42,7 +42,7 @@ them. hook can see but cannot interpret. A payload from which no result text can be obtained is a different thing — there is nothing to interpret — so fail-open's rationale does not reach it. Note the narrower premise this rests on: for the *pinned* server the shape is - unproducible, but a mapped third-party tool may legitimately return empty or non-text + unproducible, but a third-party tool may legitimately return empty or non-text content (§3.1). Fail-closed holds either way, because an unreadable result is not evidence of a review; only the diagnosis differs (§6). 4. **A discarded pass reports the fact on every occurrence; the setup advice appears once @@ -97,6 +97,18 @@ string is never a candidate. **The matcher consumes the payload's own bytes** — the located span, copied out with nothing decoding, normalizing or re-encoding it in between. +**And so does the selection, which is a KNOWN GAP in this section's own contract.** "First +`text` element" above is a *semantic* description, but the implementation compares the `type` +value as raw bytes. A Unicode-escaped spelling of `text` is legal JSON that means `text`, and +it is **not** selected: with no other element the locator reports nothing usable and the class +is `no-result`. That is fail-closed — a real result is discarded rather than miscounted, which +is the safe direction — but it is still a wrong verdict on a legal payload, and the same +applies to any escaped spelling of the `type` and `text` keys. It is stated here rather than +quietly narrowed because the sentence above would otherwise promise semantic matching the code +does not do. Characterized by a regression row, carried in `todos.md` with a trigger; closing +it means decoding the `type` value for equality while still returning the selected `text` in +its original escaped bytes, since the matcher downstream depends on those. + **Mislocation is safe by construction, and that is the locator's whole contract.** Every state the scan cannot resolve — a repeated depth-1 key, a structure it cannot walk, a value that is not the settled shape — returns *cannot determine* rather than a guess, and the @@ -116,8 +128,14 @@ respectively. **Why fail-closed is right here, at the precision the evidence supports.** For the pinned server these shapes are unproducible, so each indicates a payload-contract change. That -premise does **not** generalize: a mapped third-party tool may legitimately return empty -content, an image block, or another non-text shape. Fail-closed holds either way — an +premise does **not** generalize: a third-party tool may legitimately return empty +content, an image block, or another non-text shape. **Such a tool reaches the gates by two +routes, and naming only one is what this paragraph exists to prevent:** through a mapping in +`.context/codex-gate.tools`, *or* as a server registered under the default name `codex` — +which is decision 1's own remedy, so it is the expected configuration rather than an exotic +one. An absent mapping therefore does not rule a third-party tool out, and no diagnosis may +assume it does. **This paragraph is the single normative statement of that boundary; §6's +messages and every other mention derive from it rather than restating it.** Fail-closed holds either way — an unreadable result is not evidence of a review — but the *diagnosis* must not assume the harness is at fault (§6). @@ -171,6 +189,18 @@ that might have been a `Bash` PreToolUse. A payload the hook cannot **route** th touches no state and emits nothing, exiting 0 — the pre-existing behaviour for anything it cannot parse, and unchanged here. +**"Cannot route" is narrower than "is malformed", and the gap belongs to `field()` rather +than to classification.** Routability depends on whether `jq` is present, and that predates +this change: **with** `jq`, `field()` on a malformed document returns empty and the hook +exits silently, so the paragraph above holds as written. **Without** `jq`, the `grep` +fallback can still read a `tool_name` out of a document that is malformed elsewhere, so the +same payload *does* route — and then classifies normally, landing in `unrecognized` if the +locator refuses it, which counts and discloses. So a malformed payload is not uniformly +inert: it is inert when unroutable, and an ordinary counted-and-disclosed `unrecognized` +when the fallback routes it. That divergence is `field()`'s and is deliberately left alone; +it is stated because a reader comparing the two environments would otherwise read it as a +classifier defect. + **One outcome is settled here**, because it is a classification rather than scanner mechanics: a **repeated depth-1 `tool_response` key** is **`unrecognized`** (the locator is genuinely ambiguous). The plan defines only how the scanner *recognizes* that state, not what it means. @@ -199,6 +229,18 @@ such as `still running after` is *not* the anchor: it can legitimately occur ins result's own summary — a review discussing this mechanism would contain it — and would let one payload satisfy two anchors. +**The segment must also fall within the first 4096 units of the block**, and that bound +NARROWS this class rather than merely implementing it. A notice whose quoted tool name is +long enough to push the segment past the cutoff is `unrecognized`, so it **counts** instead +of being discarded — the wrong direction, and stated here because §4 otherwise reads as +covering *any* mapped name at *any* length. The bound exists because the unbounded form was +quadratic: `${b%%\n*}` in bash 3.2 evaluates by trying successively longer suffixes, and an +anchor-prefixed block that never completes the notice cost 5.9 s at 150 KB in one +synchronous hook invocation. 4096 is roughly 200x the longest real tool name, and the +boundary is tested immediately inside and outside it rather than assumed. Raising or +removing it requires a non-quadratic way to find the first newline, not just a bigger +number. + **Precedence, stated defensively:** `backgrounded` is tested before the envelope polarity, and only at start-of-text. A genuine notice never begins with an encoded envelope opening and a genuine envelope never begins with the notice prose, so the two are disjoint by @@ -269,7 +311,7 @@ in the suite clears them, or one-shot tests become order-dependent. | absent | `unrecognized`, gate on, emit succeeds | `unverified` written | | absent | `unrecognized`, gate on, **emit fails** | `unverifiedPending` written (best-effort) | | absent | `unrecognized`, gate off (emit suppressed) | `unverifiedPending` written | -| pending | any unsuppressed hook event | disclosure emitted, then `unverified` written and pending cleared | +| pending | any unsuppressed **routed** hook event | disclosure emitted, then `unverified` written and pending cleared | | pending | emit fails, or `unverified` write fails | pending **retained** | | shown | `unrecognized` again | nothing emitted, nothing written | @@ -297,7 +339,7 @@ is why it is accepted rather than engineered around. **Pending is not only about the off-switch.** A failed emit while the gate is *on* leaves the same debt: the call is counted and the disclosure was never delivered. Without a pending -write there, an unrelated later event has nothing to flush, and the workspace can reach a +write there, an unrelated later routed event has nothing to flush, and the workspace can reach a satisfied count made of uninspected calls with the once-per-workspace disclosure never shown — which is decision 2's guarantee broken through a path the gate-off reasoning never covered. So a failed emit takes the same best-effort pending write as suppression does. @@ -334,7 +376,7 @@ cannot be two emits. When an invocation owes both a pending disclosure and a per message, they are **composed into one emit** — disclosure first, then the per-occurrence message — with both `additionalContext` bodies joined and both `systemMessage` bodies joined. Nothing is dropped and nothing is deferred; a deferred message would collide again -on the next event. +on the next routed event. **Field split.** `additionalContext` (model-facing) carries the consequence for the gate. `systemMessage` (user-visible) carries any operator action, because the operator is who can @@ -357,8 +399,9 @@ perform it. - **`no-result`, every occurrence.** No tool result was obtainable, so the pass was not counted. It names **two** causes rather than assuming the harness is at fault: a hooks-API payload contract change (check the Claude Code version and report it — for the pinned - server this shape is unreachable), **or** a mapped third-party tool returning empty or - non-text content, which is legitimate for that tool and simply unreadable as a gate result. + server this shape is unreachable), **or** a third-party tool returning empty or + non-text content — reaching the gates either through a mapping or as a server registered + under the default name `codex`, so an absent mapping does not rule it out — which is legitimate for that tool and simply unreadable as a gate result. **The tool name alone cannot tell them apart**, and the message must not pretend otherwise: decision 1's own remedy is registering a third-party server *as* `codex`, which makes the names identical. The checks it gives instead are `.context/codex-gate.tools` for @@ -372,7 +415,9 @@ perform it. - **a reworded harness backgrounding notice** (§4) — the call was backgrounded and the anchor no longer matches, so the remedy is the same `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` guidance the `backgrounded` message carries; - - a mapped third-party tool whose envelope this hook cannot read — **no user-side fix**; + - a third-party tool whose envelope this hook cannot read — mapped, or registered under + the default `codex` name, since decision 1's own remedy makes the names identical — + **no user-side fix**; - a payload the locator refuses as ambiguous (§3.1) — installing or removing `jq` does **not** change this, since classification never consults it; - a hook parser defect — **no operator fix**; the check is to run the hook against the diff --git a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md index f72948b..2205942 100644 --- a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md +++ b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md @@ -100,8 +100,13 @@ setting: a discarded pass reads as an actionable setup gap, not as a failed revi `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` as the fix. - [ ] Given a payload from which no result text can be obtained, no counter advances and no fingerprint is stored, and the reminder names the causes that actually apply — a - hooks-API payload contract change, or a mapped tool returning empty or non-text - content — rather than diagnosing backgrounding. + hooks-API payload contract change, or a third-party tool returning empty or non-text + content — which reaches the gates either through a mapping in `.context/codex-gate.tools` + or as a server registered under the default name `codex`, so an absent mapping does not + rule it out — rather than diagnosing backgrounding. + (Amended 2026-08-02 at Gate-B pass 5: the shipped prompt was corrected for this at + P9-33 and the criterion was left naming only the *mapped* case, so the story and the + prompt disagreed about the trust boundary.) (Amended 2026-07-31, during the design's Gate-A pass 4. The first two criteria originally read as one: "a payload carrying no readable Codex result at all — the auto-backgrounding notice being the observed instance". That was true while those were @@ -117,9 +122,26 @@ setting: a discarded pass reads as an actionable setup gap, not as a failed revi disables auto-backgrounding rather than forcing it. - [ ] Regression tests cover shapes 1–3 and the success path, and the shape-1 and shape-2 tests fail against the pre-change hook. -- [ ] The probe methodology is re-run against the changed hook and its counter readings +- [ ] ~~The probe methodology is re-run against the changed hook and its counter readings recorded: shapes 1–2 from the captured payloads, shape 3 with the variable absent, - shape 3 with it set, and one genuine pass that still counts. + shape 3 with it set, and one genuine pass that still counts.~~ + **Amended 2026-08-02, confirmed by Daniel.** Satisfied by a guarded replay of the + captured payloads through the changed hook instead of a live re-run: 9/9 rows, each + asserting fixture-read status, `sed` status *and* effect, the routed tool name and + the exact class against an explicit per-row expectation, with an exact row-count + check — and the guards themselves negative-checked in three directions (wrong + expected class, missing fixture, wrong row count) before the result was accepted. + **Why the live re-run was dropped rather than deferred:** it requires instrumenting + the *installed* hook, which serves every concurrent Claude Code session on this + machine, and Task 1 established that no isolated profile can be driven from this + session. The two variable states it would have distinguished are already settled + elsewhere and not by this criterion: shape 3 was confirmed live during Task 1 (a + backgrounded call delivered its result by task notification and fired **no** second + `PostToolUse`), and the variable-set path is the harness's, not the hook's — with it + set the notice never arrives, so there is no hook behaviour left to observe. What + the amendment gives up is stated rather than glossed: no live evidence that the + *changed* hook meets a real backgrounding notice in situ, only that it classifies + the captured one correctly. - [ ] The plugin manifest version is bumped. ## 4. Affected AGENTS.md invariants diff --git a/plugins/dev-workflow/.claude-plugin/plugin.json b/plugins/dev-workflow/.claude-plugin/plugin.json index 90268f8..654ea02 100644 --- a/plugins/dev-workflow/.claude-plugin/plugin.json +++ b/plugins/dev-workflow/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "dev-workflow", "displayName": "Cross-Model Review Workflow", - "version": "0.7.1", + "version": "0.8.0", "description": "Spec-driven workflow with two independent cross-model review gates, an append-only hardening ledger with an escalation ladder, and repo-enforced quality. Requires the superpowers plugin.", "author": { "name": "Daniel Sänger", diff --git a/plugins/dev-workflow/CHANGELOG.md b/plugins/dev-workflow/CHANGELOG.md index 8fc08fb..d011622 100644 --- a/plugins/dev-workflow/CHANGELOG.md +++ b/plugins/dev-workflow/CHANGELOG.md @@ -22,6 +22,109 @@ unambiguously, still fails. Deleting only a plugin's *manifest* while the direct keeps shipping fails too. AGENTS.md invariant 12 carries the complete list. +## 0.8.0 + +- **The gate hook reads the result of a gate call before counting it.** Until now it + counted on tool name alone, so a Codex call that failed, timed out, or was auto- + backgrounded at 120 s advanced the counter — and for Gate B stored a fingerprint over + content nobody read, which is a false ✓ in the hook's own recorded state. Observed + directly: `passCountA` moved 3 → 4 on a call that ran 272 ms and never started a + review. The pinned `mcp-codex-dev` catches its own errors and returns them as ordinary + results carrying `success: false`, so Claude Code sees a successful tool call; nothing + in the hook looked further. Five classes now decide it — `success` and `unrecognized` + count and store, `failure`, `backgrounded` and `no-result` do neither. An escape-aware + scan (never `jq`, which would reserialize away the very escape variants the matcher + reads) locates the first `text` element of `tool_response`; the failure marker must be + the envelope's **immediately-first** property, so a reordered envelope degrades to + `unrecognized` rather than to a wrong verdict. +- **Fail-open where locating is uncertain, fail-closed where it is certain.** An + unambiguous "there is nothing usable here" is `no-result` and does not count. Not being + able to determine anything — unwalkable structure, a repeated `tool_response` key, a + payload past the 1 Mi-unit scan bound or the 200-frame depth cap — is `unrecognized`, + which **counts**, with a once-per-workspace disclosure saying the count was made + without inspection. Counting an uninterpretable result silently was the alternative, + and it is the direction invariant 2 names as dangerous. +- **The counter is closer to the truth and is still not evidence.** Classification cannot + see whether the findings file was written, so §5's rule is unchanged and now stated + with its reason: discount every incomplete pass regardless of what the counter says. + `CLAUDE.md`, `README.md` and the `/workflow-init` inline template were corrected where + they taught the old tool-name-only mechanism. +- **Fixes an invariant-1 violation that shipped.** With a directory at a marker path, the + old `: > "$file"` form exits **2** under `dash` while exiting 0 under macOS `sh` — so + the hook could fail non-zero on Linux, which invariant 1 forbids. The existing + regression test never caught it because it only ever ran under macOS `sh`. +- **The test suite now really runs the hook under both shells.** Every runner used to + invoke the hook as `sh`/`/bin/sh` regardless of what ran the suite file, so running it + with `dash` exercised the *harness* under dash and the hook under whatever `/bin/sh` is + — bash, on macOS. Only a handful of explicit `dash` rows ever reached dash, and this + release's first draft generalized from them to the whole suite. `HOOK_SH` now selects + the shell the hook itself runs under, CI runs the suite twice, and the claim is true + rather than corrected. +- **The common result-scan paths are bounded, and timed regression rows keep them that + way.** Three quadratics made a large-but-legal result stall the synchronous hook: the + locator built each string byte-by-byte (and `substr(s,j,1)` is O(len) per call in BWK + `awk`, so any per-character walk is quadratic by itself); the `backgrounded` test ran + `${b%%\n*}` on every block, which bash 3.2 evaluates by trying successively longer + suffixes; and guarding that expansion on the notice's literal prefix still left it + running in full for a block that starts with the anchor and never completes it. A + 150 KB text block — ordinary for a Gate-B review result, and a seventh of the 1 Mi-unit + ceiling — took **10.9 s** and now takes **0.46 s**; the anchor-prefixed near miss took + 5.9 s and now takes under a second, at the cost of one stated limit (the notice segment + must fall within the first 4096 characters, so a ~4070+ character tool name counts + instead of being discarded). + **Not fixed, and named rather than left to be found — two paths.** `skipval` still walks + containers character by character, so a large *valid sibling container before* + `tool_response` costs 3.2 s at 200 KB and 11.5 s at 400 KB; and the record accumulator + rebuilds the whole input once per input line, so a newline-rich payload is quadratic in + line count independently of that (0.35 s at 4k lines, 2.69 s at 16k). Only the 1 Mi-unit + ceiling stops either, and a payload just under it still costs tens of seconds. The lesson is written into + the code: a size backstop bounds work only if the per-byte cost is constant, and in + POSIX shell and `awk` it often is not — and a timed regression row only covers the + branch its own fixture reaches. +- **`.context/codex-gate.off` is not a rollback**, and the docs no longer imply the + counters are "accurate" while it is set. It silences messages; classification and state + tracking keep running, so re-enabling lands on counters with the same semantics as + gate-on — which is not the same as evidence that a review happened. +- **Mapping a Codex tool now says the one thing that made it fail silently, and the + parser enforces it.** A mapped name must lie in `mcp__codex__*`, because the hook's + matcher is `^(Bash|Skill|mcp__codex__.*)$`, so an out-of-namespace name is either never + delivered at all or — for the two reserved names it does deliver — hijacks a lifecycle + event. Two remedies that told operators to rename the server *away* from `codex` are + removed — they produced exactly that unreachable configuration. **And the two reserved + names that DO fire were a false-✓ hazard:** the mapped cases are tested before the + native `Bash` and `Skill` cases, so `reviewTool=Bash` made a `git commit` **count** a + Gate-B pass instead of resetting the cycle, and `execTool=Skill` counted a skill + invocation as a Gate-A pass — both reachable from a plausible typo. Out-of-namespace + mappings are now ignored like any other unusable line. +- **New setup step: `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`** (README), the primary defence + against auto-backgrounding. Requires Claude Code 2.1.212+, must be set in the + environment Claude Code is launched from, `0` disables backgrounding and a positive + value must exceed your longest gate call. + +**Four accepted residuals, named rather than left to be discovered.** Each is recorded +because a reader who assumes otherwise will trust the counter further than it earns. + +- **C1** — the `backgrounded` class recognizes the harness notice **in the wording it + currently uses**. On any runtime where auto-backgrounding is still effective and that + prose has changed, the call is counted again — as `unrecognized`, with the disclosure. + This is why the environment variable is the defence and the hook is the backstop. +- **C2** — an `unrecognized` call whose disclosure is neither delivered nor persisted is + counted **silently**, in both directions: emit suppressed (gate off) and emit failed, + each combined with a failed pending write. Two named oracles pin it rather than repair it. +- **C3** — counter mutation is unserialized and `.context/` is trusted. Pre-existing, + filed in `todos.md`; no locking is added here. +- **C4** — concurrent check-emit-write on the diagnostic markers can duplicate or lose a + disclosure, in both directions. Spec §6 accepts this under "delivery is best-effort". + The sequential tests describe sequential behaviour and must not be read as guaranteeing + more; a partial fix over one state family would be the inconsistent repair C3 refuses. + +**Rolling back to 0.7.1 restores the original defect**, and that is the whole trade: failed, +timed-out and backgrounded calls count as gate passes again, and the `dash` special-builtin +exit returns with them — on Linux an unwritable `.context/` makes 0.7.1's hook exit 2, +violating invariant 1. So roll back if 0.8.0 discards passes it should count, and not for +anything else. What it does **not** undo: the three diagnostic markers already written into +`.context/` stay there, and 0.7.1 ignores them. + ## 0.7.1 - **Gate B gains a standing lens: "which existing statements does this diff falsify?"** diff --git a/plugins/dev-workflow/commands/workflow-init.md b/plugins/dev-workflow/commands/workflow-init.md index 71f008c..ee704d1 100644 --- a/plugins/dev-workflow/commands/workflow-init.md +++ b/plugins/dev-workflow/commands/workflow-init.md @@ -71,8 +71,13 @@ Check, in order: only a **local**-scope `codex` can shadow this project's entry; restarting will never change that. Fix: confirm which entry actually wins with `claude mcp get codex`, then remove that one by its real scope — - `claude mcp remove codex -s `. Alternatively rename the project entry and - mirror the new name in `.context/codex-gate.tools`. + `claude mcp remove codex -s `. **Renaming the project entry is not an + alternative:** the hook is invoked by a `hooks.json` matcher of + `^(Bash|Skill|mcp__codex__.*)$`, so a server registered under any other name puts + its tools outside that namespace, where the hook is never invoked for them — + `.context/codex-gate.tools` cannot map that back, because mapping is read by a hook + that never runs. Whatever server is meant to back the gates must end up registered + as `codex`. **A *user*-scope `codex` is NOT this cause.** User scope loses to project scope, so a user entry alongside a project entry that is merely unapproved is cause 1 — @@ -121,7 +126,7 @@ Check, in order: switch to the pinned mcp-codex-dev (Step 2.8), or map the names in .context/codex-gate.tools: execTool=mcp__codex__codex - reviewTool= + reviewTool=mcp__codex__ ``` Prefer switching servers over mapping: a mapping can only be honest if the server @@ -129,6 +134,12 @@ Check, in order: one general-purpose tool makes the counters move without either gate meaning what it claims — a false ✓, which is worse than the STOP it silences. + **A mapped name must also lie in the `mcp__codex__*` namespace.** The hook is invoked + by a `hooks.json` matcher of `^(Bash|Skill|mcp__codex__.*)$`, so a mapping naming a + tool outside it is either never delivered — the mapping looks applied and does nothing — + or, for the reserved names `Bash` and `Skill`, is delivered and hijacks a lifecycle + event. The hook refuses both. (Normative statement: the design's decision 1.) Register the server under the name `codex` to place its tools there. + With no Codex reachable, **both review gates are inoperative** — the single most important thing this command can tell the user. If the user chooses not to set it up now, switch to the degraded mode in Step 2.13 rather than scaffolding gates that @@ -153,9 +164,10 @@ Prerequisites: claude plugin install superpowers@superpowers-marketplace codex MCP NOT LOADED — `claude mcp get codex` shows a local-scope entry winning over this project's, so exec/review never - load. Restarting will not help: remove the winning - entry (claude mcp remove codex -s local), or rename - this one and mirror it in .context/codex-gate.tools + load. Restarting will not help, and renaming this + entry would move its tools out of mcp__codex__* + where the hook is never invoked: remove that entry + (claude mcp remove codex -s local) gh CLI ok (optional — only /process-pr-review needs it) AGENTS.md absent — Step 3 will write it stack pnpm · TypeScript · vitest · GitHub Actions @@ -248,9 +260,14 @@ Independent second opinion at two gates. Easiest steps to skip, so the disciplin yours — a non-blocking hook (shipped by the `dev-workflow` plugin) reminds you at each. Opt out per-workspace with `.context/codex-gate.off` (delete to re-enable); the gates still apply. The hook counts passes by TOOL NAME (`mcp__codex__exec` / -`mcp__codex__review`); if your Codex MCP server names them differently, map it in -`.context/codex-gate.tools` (`execTool=` / `reviewTool=`) — otherwise your -reviews are invisible to the counters and Gate B reports "not run" forever. +`mcp__codex__review`) and by RESULT ENVELOPE — it withholds the count for a routed gate +call whose result it reads as failed, backgrounded, or yielding no usable text. If your +Codex MCP server names its tools differently, map it in `.context/codex-gate.tools` +(`execTool=` / `reviewTool=`) — otherwise your reviews are invisible to the +counters and Gate B reports "not run" forever. A mapped name must itself start with +`mcp__codex__`, or it is refused — outside that namespace it is either never delivered +to the hook or, for `Bash`/`Skill`, hijacks a lifecycle event. Register the server as +`codex` to place its tools there. **Both gates are a LOOP with a HARD FLOOR: min 3 passes per run (Blocker/Major only), counted by the hook.** The hook counts passes but can't read findings or @@ -344,17 +361,27 @@ write or just returns its prior summary is not established; if it returns the su that was the attempt. Spent and still incomplete → STOP and surface, naming which check failed. -**What this does not do.** The hook counts on `PostToolUse`, keyed on tool name, and -never sees the file. Claude Code fires `PostToolUse` after a *successful* call and routes -a failed one to `PostToolUseFailure`, which the plugin registers no handler for — but do -not infer from that which failures escape counting: the pinned `mcp-codex-dev` catches -its own errors, executor timeouts and aborts included, and returns them as a normal -result carrying `success: false` rather than throwing or setting `isError`. A failed -review therefore looks like a successful tool call and increments the counter. So does a -call that returns and then fails validation. The rule that follows is the simple one: -**discount every incomplete pass regardless of what the counter says** — a "satisfied" -count can overstate the passes you actually hold, and reasoning about which failure took -which event path will get it wrong. Nothing checks the terminator mechanically; this is +**What this does not do.** The hook counts on `PostToolUse`, keyed on tool name **and on +the result envelope**, and still never sees the file. Claude Code fires `PostToolUse` +after a *successful* call and routes a failed one to `PostToolUseFailure`, which the +plugin registers no handler for — but do not infer from that which failures escape +counting: the pinned `mcp-codex-dev` catches its own errors, executor timeouts and aborts +included, and returns them as a normal result carrying `success: false` rather than +throwing or setting `isError`. A failed review therefore still looks like a successful +*tool call* — but as of 0.8.0 the hook reads the result of gate calls it can route, and +withholds the count for three **recognized** shapes: an envelope whose **first** property +is `success: false`, the harness backgrounding notice **in the wording it currently +uses**, and a result from which no usable text can be obtained. Every other routed gate +call counts, including any located text the hook cannot interpret — a reordered envelope, +a reworded notice, an unknown third-party shape — which counts **with** a disclosure that +is attempted and normally shown once per workspace, but can be lost or repeated when its +marker cannot be persisted. So does a call that returns and then fails validation. The +counter is therefore closer to the truth than it was and is still not evidence: a +"satisfied" count can still overstate the passes you actually hold, and reasoning about +which failure took which event path will get it wrong. The rule that follows is the +simple one: **discount every incomplete pass regardless of what the counter says**, +because classification cannot see whether the findings file was written. +Nothing checks the terminator mechanically; this is instruction-backed by design, and a recurring truncation incident is the trigger to build the checker, not a reason to build it now. Detection is conditional: it catches an absent or malformed terminator, a count mismatch and a missing branch file *in the artifact you @@ -568,11 +595,14 @@ like the rest of §5; the detection is a reader comparing the pass against the s - **Timeout / abort:** a codex call that dies at the MCP tool-call timeout is retried once before surfacing to the user, and that retry *is* the single shared recovery attempt above — not a second one. An abort is an incomplete pass, so treat it as one: - it may already have moved the hook's counter (the pinned server returns its own - timeouts as ordinary results), and it may have left a partial or stale target file, so - delete the targets and confirm them gone before retrying, then validate the result like - any other pass. Counter and workspace state persist in `.context/`; the *pass* does - not. + it may have left a partial or stale target file, so delete the targets and confirm them + gone before retrying, then validate the result like any other pass. Whether it moved the + hook's counter depends on the shape it returned and on which hook version is installed: + as of 0.8.0 a recognized failure envelope, the recognized backgrounding notice and a + result yielding no usable text are all withheld from the count, while a reordered, + reworded or unrecognized shape still counts fail-open. Do not reason from the counter + either way — an incomplete pass is discounted whatever it says. Counter and workspace + state persist in `.context/`; the *pass* does not. --- @@ -1034,8 +1064,9 @@ gate instructions that cannot run, plus a Gate-B STOP on every single commit for is noise that trains the user to ignore the hook, and a hook people ignore is worse than no hook. Instead, degrade explicitly: -1. Write `.context/codex-gate.off` so the hook stays silent (it keeps tracking state, - so re-enabling later is accurate rather than stale). +1. Write `.context/codex-gate.off` so the hook stays silent (it keeps classifying and + tracking state, so re-enabling later lands on counters carrying the same semantics + as gate-on — never evidence that a review happened). 2. Add one line at the very top of §5 in the scaffolded `CLAUDE.md`: ```markdown diff --git a/plugins/dev-workflow/hooks/codex-gate.sh b/plugins/dev-workflow/hooks/codex-gate.sh index e57c31a..396bd44 100755 --- a/plugins/dev-workflow/hooks/codex-gate.sh +++ b/plugins/dev-workflow/hooks/codex-gate.sh @@ -50,6 +50,13 @@ noted_file="$state_dir/codex-gate.toolNote" # marks the unknown-tool note count_file="$state_dir/codex-gate.passCount" # Gate B (review) passes since last commit fresh_file="$state_dir/codex-gate.freshCount" # Gate B passes covering the CURRENT tree countA_file="$state_dir/codex-gate.passCountA" # Gate A (exec) passes since last plan execution +# Diagnostic markers. These are NOT gate-pass state: they dedupe one-time disclosures, +# and every write is best-effort. `bgAdvice` is independent of the other two; +# `unverified` means the uncertainty disclosure was shown, `unverifiedPending` that it is +# owed because the flush that would have shown it was suppressed or failed. +bgadv_file="$state_dir/codex-gate.bgAdvice" +unver_file="$state_dir/codex-gate.unverified" +pend_file="$state_dir/codex-gate.unverifiedPending" # FINDING G: the plugin is installed globally, but the workflow is adopted per project. # A repo that never ran /workflow-init has no gate to enforce, so the hook does NOTHING @@ -134,14 +141,24 @@ if [ -f "$tools_file" ]; then while IFS='=' read -r k v || [ -n "${k:-}" ]; do # Trim the EDGES only. Deleting all whitespace would rewrite `execTool=has space` # into the perfectly valid name `hasspace` and honor it — turning a typo into a - # gate pointed at a tool that never fires, which is the failure this parse guards. + # gate pointed at a tool the hook is never invoked for, which this parse guards. trim='s/^[[:space:]]*//; s/[[:space:]]*$//' k=$(printf '%s' "${k:-}" | sed "$trim") v=$(printf '%s' "${v:-}" | sed "$trim") # Same rigor as the floor file: only a plausible tool name is honored. Anything # else — empty, a comment, a glob character, an unknown key — is ignored, so a - # typo'd mapping can't silently point a gate at a tool that never fires. + # typo'd mapping can't silently point a gate at a tool the hook never sees. case "$v" in '' | *[!A-Za-z0-9_-]*) continue ;; esac + # ...and it must lie in the `mcp__codex__*` namespace, which is a CORRECTNESS check + # and not merely consistency with the documented contract. `hooks.json` matches + # `^(Bash|Skill|mcp__codex__.*)$`, so a name outside that namespace either never + # fires — a mapping that looks applied and does nothing — or, for the two reserved + # names that DO fire, hijacks them: the mapped cases below precede the native `Bash` + # and `Skill` cases, so `reviewTool=Bash` made a `git commit` COUNT a Gate-B pass + # instead of resetting the cycle, and `execTool=Skill` counted a skill invocation as + # a Gate-A pass. Both are false ✓ in recorded state — the direction invariant 2 calls + # dangerous — reachable from a plausible typo. Ignored like any other unusable line. + case "$v" in mcp__codex__?*) ;; *) continue ;; esac case "$k" in execTool) exec_tool="$v" ;; reviewTool) review_tool="$v" ;; @@ -288,22 +305,447 @@ emit() { # $1 = additionalContext (model-visible), $2 = systemMessage (user) # supported channel for a non-blocking reminder the model must actually read. # # Per-workspace opt-out: while .context/codex-gate.off exists, stay silent. - # State tracking (SET/INVALIDATE/RESET) keeps running so re-enabling is accurate. + # State tracking (SET/INVALIDATE/RESET) keeps running while off, so re-enabling + # carries the same counting semantics as if the gate had been on — not a guarantee + # that every counted call was reviewed. # - # Returns 1 when suppressed, 0 when something was actually written — callers that - # dedupe a one-time note key their marker off that, so a note suppressed here is - # still available once the workspace opts back in. + # THREE statuses, and callers must treat ONLY 0 as "shown": + # 0 — a complete hook JSON document was written + # 1 — suppressed by the off-switch + # 2 — the write failed + # A one-time note keys its marker off 0 alone. Returning 0 after a failed write burns + # the one-shot on a message nobody can read, and the note never comes back. [ -f "$off_file" ] && return 1 if command -v jq >/dev/null 2>&1; then # jq encodes the strings, so any character (incl. control chars) is escaped correctly. jq -cn --arg ev "$event" --arg ctx "$1" --arg msg "$2" \ - '{hookSpecificOutput:{hookEventName:$ev,additionalContext:$ctx},systemMessage:$msg}' + '{hookSpecificOutput:{hookEventName:$ev,additionalContext:$ctx},systemMessage:$msg}' || return 2 else # Fallback (no jq): escape backslash + quote. The supported reminders are # static and control-char-free, so this is sufficient. - ctx=$(printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g') - msg=$(printf '%s' "$2" | sed 's/\\/\\\\/g; s/"/\\"/g') - printf '{"hookSpecificOutput":{"hookEventName":"%s","additionalContext":"%s"},"systemMessage":"%s"}\n' "$event" "$ctx" "$msg" + # + # BOTH substitutions are status-checked. A failed one yields an EMPTY field while + # `printf` still exits 0, so without the check a document reading + # `additionalContext:""` would be written and reported as a successful write — + # well-formed, empty, and having spent the one-shot. + # `$event` is escaped TOO, and it is the one field that was not. It comes from the + # payload, so it is untrusted: an event name ending in a backslash emitted + # `"hookEventName":"Bogus\","`, where the backslash escapes the closing quote and + # Claude Code receives invalid JSON. The jq branch above never had this — `--arg` + # encodes it — so the defect existed only on the fallback path. Routing now also + # refuses to flush an unroutable event, which closes the only route that reached + # here with a hostile name; this escape is the second of the two, kept because a + # field interpolated raw next to two escaped ones is a trap for the next editor. + ev=$(printf '%s' "$event" | sed 's/\\/\\\\/g; s/"/\\"/g') || return 2 + ctx=$(printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g') || return 2 + msg=$(printf '%s' "$2" | sed 's/\\/\\\\/g; s/"/\\"/g') || return 2 + printf '{"hookSpecificOutput":{"hookEventName":"%s","additionalContext":"%s"},"systemMessage":"%s"}\n' "$ev" "$ctx" "$msg" || return 2 + fi + return 0 +} + +# --- RESULT CLASSIFICATION MESSAGES ------------------------------------------------- +# Target model: Claude Sonnet 4.5 / Opus 4.1 via Claude Code (prompting guidance checked +# for that model family). FIVE PAIRS, ten strings, all declared unconditionally so `set +# -u` cannot abort on any path. +# +# The field split follows spec §6: `additionalContext` is read by Claude via Claude Code, +# `systemMessage` by the operator. A remedy only a human can perform — restarting Claude +# Code, editing a config, changing a server timeout, unmapping a tool — belongs in +# `systemMessage`, because the model receiving `additionalContext` cannot do any of it. +# +# Each is a SINGLE-QUOTED shell string, so no ASCII apostrophe may appear inside one: +# POSIX shell cannot escape an apostrophe within single quotes. Inner examples use double +# quotes and possessives are phrased around it. +FAILURE_CTX='Claude via Claude Code — gate hook. this Codex call returned an envelope reporting failure. Not counted as a gate pass, no review fingerprint stored, does not count toward the floor. Read error.code in the tool result. CODEX_EXECUTION_FAILED is the pinned server generic failure code and does NOT tell you whether the call started, so check the accompanying error message and any session artifacts before assuming nothing ran; a call that did start may have left work behind. CODEX_TIMEOUT means the executor gave up mid-run: re-run the SAME call with the SAME scope. Any other code, or no code at all, is unclassified: this state covers every envelope the hook managed to route AND read as carrying success false as its first property, which is not limited to the two codes named here — routing, locating the result text and the raw key spelling each gate it, and an envelope failing any of those lands in a different state instead. So re-run once with the same scope and, if it repeats, report the code and message verbatim together with the effective server name and version from claude mcp list — an unfamiliar code is itself evidence about which server answered. Never retry with a narrower instruction or a smaller range, because that would count a pass for less than the artifact or diff the gate requires. Report one line: "gate pass discarded | error-code | started yes/no/unknown". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the retry also fails, stop and surface that line; the operator note carries the configuration remedy.' +FAILURE_MSG='⚠ Codex call failed — not counted as a gate pass. If the code was CODEX_TIMEOUT, the fix is configuration and only you can apply it: raise the executor timeout for the Codex MCP server, or reduce load outside the review. Do not ask for a smaller review scope — a narrower pass is worth less than a slow one.' + +NORESULT_CTX='Claude via Claude Code — gate hook. this gate call carried no result text the hook could read. Not counted as a gate pass, no review fingerprint stored. Treat the pass as not run and report it. Two causes produce this shape and the tool name cannot separate them: a hooks-API payload change, or a third-party tool returning empty or non-text content — which reaches the gates either through a mapping in .context/codex-gate.tools or as a server registered under the default name codex, so an absent mapping does not rule it out. Report one line: "gate call unreadable | mapped yes/no from .context/codex-gate.tools | claude-code version". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. One retry per pass is the whole recovery budget under CLAUDE.md section 5. A repeat is configuration or contract, not a transient failure, so stop and surface it; the operator note carries both checks.' +# shellcheck disable=SC2016 # the backticks quote a command for a human to read, verbatim +NORESULT_MSG='⚠ Gate call returned no readable result — not counted. Run both checks before concluding. First: does .context/codex-gate.tools map a tool name? Second: what does `claude mcp list` show as the effective server and version — not what .mcp.json says, because scope precedence can make a different entry of the same name effective. These checks narrow the cause; they do not prove it. If a mapping or a third-party server is in play, that tool may be returning empty or non-text content, which it can do legitimately: unmap it, or replace it with a server exposing exec and review. If both checks show the pinned server at its pinned version, a payload-contract change is the remaining explanation — record your Claude Code version and report it.' + +BG_LONG_CTX='Claude via Claude Code — gate hook. this gate call was moved to the background at the auto-background threshold, 120 s by default, so its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. The original call may still be running and can still write its findings file later. If the tool result carries a task id, stop that task by it; if it carries none, wait for the call to finish. Do that before deleting that slot or re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. Do not re-run while that task is active: a late writer landing in a slot you already re-ran leaves a correctly terminated file from the wrong run, and no downstream check can detect that. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it as a setup problem rather than retrying again, reporting one line: "gate pass discarded | backgrounded | second occurrence | CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS set yes/no".' +BG_LONG_MSG='⚠ Gate pass discarded (backgrounded) — a setup gap, not a failed review. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from, then restart Claude Code: it reads the value at process start, so exporting it inside a tool shell leaves the running session unchanged. Use 0 to disable auto-backgrounding, or a positive value that exceeds your longest gate call, since a positive value shorter than the call still backgrounds it. Requires Claude Code 2.1.212 or newer.' + +BG_SHORT_CTX='Claude via Claude Code — gate hook. this gate call was backgrounded and its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. If the tool result carries a task id, stop that task by it; if it carries none, await the original call. Do that before re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. Do not re-run while that task is active, so a late writer cannot land in a slot you already re-ran. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it rather than retrying again, reporting one line: "gate pass discarded | backgrounded | second occurrence | CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS set yes/no".' +BG_SHORT_MSG='⚠ Gate pass discarded (backgrounded) — not counted. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code; the full guidance was shown once earlier in this workspace.' + +UNVERIFIED_CTX='Claude via Claude Code — gate hook. this workspace has classified at least one gate call as countable without being able to interpret its result, and attempted to record it. The counter is a mechanical tally, not a count of completed reviews: it can include calls that failed or reviewed nothing, so it can overstate them. Judge every pass on its findings artifact and discount any incomplete or unverified call, whatever the counter says. Normally said once per workspace. It repeats when its marker cannot be persisted, when two hook runs race, or when that marker is deleted by hand or by a tool that cleans .context, so treat a repeat as a marker problem rather than as new information.' +# shellcheck disable=SC2016 # the backticks quote commands for a human to read, verbatim +UNVERIFIED_MSG='ℹ A gate call was classified as countable without inspection, and recording it was attempted. Causes with a check and a fix: a pinned-server envelope whose key order or formatting changed — compare the version in .mcp.json with the server actually serving the tools (`claude mcp list`), and pinning it back fixes it; a reworded backgrounding notice — set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code, using 0 to disable auto-backgrounding or a positive value exceeding your longest gate call, needing Claude Code 2.1.212 or newer; a broken awk or sed, which classification requires — check them functionally rather than by version flag, since --version is not POSIX and BSD sed exits nonzero for it on a healthy macOS: `printf "x\\n" | awk "{print}"` must print x and `printf "x\\n" | sed s/x/y/` must print y, each exiting 0. Causes with no user-side fix: a third-party tool whose envelope this hook cannot read — .context/codex-gate.tools names it only if it is mapped, and a third-party server registered under the default codex name reaches the gates with no mapping at all, so an absent mapping does not rule this out; where a mapping does exist, unmapping removes the gate rather than fixing the envelope; a payload the scan refused — four families land here: past the 1 Mi-unit size bound, past the 200-frame nesting-depth cap, structure the scan could not walk, and an ambiguity such as a duplicated tool_response, type or text key. Measuring the payload against those first two bounds rules them in or out; the remaining two produce this same message and the same state and are **not distinguishable from each other**, so record them as unresolved rather than guessing; and a defect in this hook parser — same situation. For either, keep the payload **locally and access-restricted**: it can contain prompts, absolute paths, review content, session identifiers and unrelated concurrent call data, so strip those before showing it to anyone, and never attach it unsanitized to a report. The list is not exhaustive: unrecognized is the terminal class, so any future unmatched shape lands here too.' + +# --- RESULT LOCATOR ------------------------------------------------------------------ +# ONE escape-aware scan locates the first `text` element of `tool_response` and yields its +# ESCAPED bytes. `jq` is NEVER used here: it would reserialize, erasing exactly the escape +# and whitespace variants the matcher reads. +# +# Statuses: 0 located (block on stdout) · 1 unambiguously nothing there · anything else +# cannot-determine. A caller must not collapse "anything else" onto a class. +# +# `locate_result` is a LOCATOR, NOT A VALIDATOR. What makes the walk trustworthy is +# string-boundary tracking and nothing else: `readstr` decides where every JSON string +# starts and ends from quote state and backslash parity alone, so a `tool_response` +# mentioned INSIDE a string is never mistaken for the key. Malformation outside a string +# boundary may be stepped over; it cannot move where the next string begins. A +# walkable-invalid document is therefore walked past and the real block located. +# +# SINGLE-QUOTED, so no ASCII apostrophe may appear anywhere below, comments included — +# one would terminate the quote and leave this hook unparseable. +# shellcheck disable=SC2016 # an awk program, not shell: $0 and $1 are awk fields +LOCATE_AWK=' +function skipws(s, i, c) { + while (i <= SLEN) { c = substr(s,i,1) + if (c==" "||c=="\t"||c=="\n"||c=="\r") i++; else break } + return i +} +# Consumes a JSON string from its opening quote. Returns the index of the closing +# quote, 0 if unterminated. RAWSTR is the RAW bytes strictly between the quotes, so +# backslash parity falls out of `esc`: a backslash takes the next byte verbatim whatever +# it is, so `\\` ends parity and the following `"` closes the string, while `\"` does not. +# +# The scan tracks parity WITHOUT building the result, then extracts once with substr. +# Appending byte-by-byte was quadratic — each append reallocates and copies a string that +# grows to the length of the value — and a 150 KB text block, ordinary for a Gate-B review +# result and well under the ceiling below, took 10.9 s in one synchronous hook invocation. +# Every byte the loop passes over is a byte of the result, none skipped or transformed, so +# the single substr is exactly what the loop used to accumulate. +function readstr(s, i, tail) { + RAWSTR="" + tail = substr(s, i+1) + if (match(tail, /^([^"\\]|\\(.|\n))*"/)) { + RAWSTR = substr(tail, 1, RLENGTH-1) + return i + RLENGTH + } + return 0 +} +# Consumes one SPAN and returns the index after it, 0 if it cannot. Not "one JSON +# value": containers keep a CLOSER STACK, so `[1}` is rejected rather than balanced by +# an undifferentiated depth counter, but the members INSIDE a balanced span are not +# parsed — `[1,]` is consumed, not refused. Primitives must be a complete JSON token; +# arbitrary bytes up to the next delimiter are rejected. +function skipval(s, i, c, st, e, j, tok) { + c = substr(s,i,1) + if (c=="\"") { e=readstr(s,i); return e ? e+1 : 0 } + if (c=="{" || c=="[") { st="" + while (i<=SLEN) { c=substr(s,i,1) + if (c=="\"") { e=readstr(s,i); if(!e) return 0; i=e+1; continue } + if (c=="{" || c=="[") { + # DEPTH CAP. The stack is a string, so each push is O(len) and deep nesting is + # quadratic in shell-visible time — work an external MCP result could dictate. + # Real payloads nest a handful deep; 200 is far above anything Claude Code emits + # and far below anything that costs. Past it: refuse, which is `unrecognized`. + if (length(st) >= 200) return 0 + st = st (c=="{" ? "}" : "]") + } + else if (c=="}" || c=="]") { + if (st=="" || substr(st,length(st),1)!=c) return 0 + st = substr(st,1,length(st)-1) + if (st=="") return i+1 + } + i++ } + return 0 } + j=i + while (j<=SLEN) { c=substr(s,j,1) + if (c==","||c=="}"||c=="]"||c==" "||c=="\t"||c=="\n"||c=="\r") break + j++ } + tok = substr(s,i,j-i) + if (tok=="true" || tok=="false" || tok=="null") return j + if (tok ~ /^-?(0|[1-9][0-9]*)([.][0-9]+)?([eE][-+]?[0-9]+)?$/) return j + return 0 +} +# Read the WHOLE input by accumulating records; the newline put back is the separator +# awk stripped. RS="\0" would be shorter and is NOT portable across POSIX awks. +# CEILING: a payload past the bound is refused rather than SCANNED. It does not bound +# memory or read work — the calling shell already holds the whole input and awk still +# ingests it — so this caps the state-machine string work and nothing else. And a SIZE +# bound only bounds that work if the per-byte cost is constant, which here it is not: +# `substr(s,i,1)` is O(len) per call in BWK awk. A 150 KB block well under this ceiling +# cost 10.9 s in one invocation until `readstr` and the `backgrounded` test were fixed. +# THIS CEILING IS LOAD-BEARING, not a formality: `skipval` still walks containers one +# character at a time, so a large valid sibling container before `tool_response` costs +# 3.2 s at 200 KB and 11.5 s at 400 KB, and this bound is the only thing that stops it — +# a payload just under it still costs tens of seconds. Do not raise it without making +# that walk cheaper first. The unit is +# awk length() units, not bytes: POSIX length() counts characters and implementations +# differ on multibyte input, so the cut-off is not identical between awks. +# Refusal is status 2, which the classifier routes to `unrecognized`: counted, disclosed. +BEGIN { MAXLEN = 1048576; over = 0; tot = 0 } +{ if (over || tot + length($0) + 1 > MAXLEN) over = 1; else { s = s $0 "\n"; tot += length($0) + 1 } } +END { + if (over) exit 2 + # ONE length() for the whole scan. Every loop below tests against SLEN rather than + # calling length(s) per iteration, and the accumulator above tracks a running total + # rather than re-measuring what it just built. + SLEN=length(s); n=SLEN; i=skipws(s,1) + if (substr(s,i,1) != "{") exit 2 + i++; found=0; first=1; closed=0 + while (i<=n) { i=skipws(s,i); c=substr(s,i,1) + if (c=="}") { i++; closed=1; break } + if (!first) { if (c!=",") exit 2; i=skipws(s,i+1); c=substr(s,i,1) } + first=0 + if (c!="\"") exit 2 + e=readstr(s,i); if(!e) exit 2 + key=RAWSTR; i=skipws(s,e+1) + if (substr(s,i,1)!=":") exit 2 + i=skipws(s,i+1) + if (key=="tool_response") { if (found) exit 2; found=1; tr=i } + i=skipval(s,i); if(!i) exit 2 } + if (!closed) exit 2 + if (skipws(s,i) <= n) exit 2 # trailing garbage after the outer object + if (!found) exit 1 + i=tr + if (substr(s,i,1)!="[") exit 1 # absent / null / non-array container + i++; firstel=1 + while (i<=n) { i=skipws(s,i); c=substr(s,i,1) + if (c=="]") exit 1 # walked the array, no text element + if (!firstel) { if (c!=",") exit 2; i=skipws(s,i+1); c=substr(s,i,1) + if (c=="]") exit 2 } + firstel=0 + if (c!="{") { i=skipval(s,i); if(!i) exit 2; continue } + j=i+1; ttype=""; ttext=""; hastype=0; hastext=0; textisstr=0; firstm=1; oclosed=0 + while (j<=n) { j=skipws(s,j); c=substr(s,j,1) + if (c=="}") { j++; oclosed=1; break } + if (!firstm) { if (c!=",") exit 2; j=skipws(s,j+1); c=substr(s,j,1) } + firstm=0 + if (c!="\"") exit 2 + e=readstr(s,j); if(!e) exit 2 + k=RAWSTR; j=skipws(s,e+1) + if (substr(s,j,1)!=":") exit 2 + j=skipws(s,j+1) + # A repeated classification-relevant member is ambiguity, not last-wins. + if (k=="type") { if (hastype) exit 2; hastype=1 + if (substr(s,j,1)=="\"") { e=readstr(s,j); if(!e) exit 2; ttype=RAWSTR; j=e+1; continue } } + if (k=="text") { if (hastext) exit 2; hastext=1 + if (substr(s,j,1)=="\"") { e=readstr(s,j); if(!e) exit 2; ttext=RAWSTR; textisstr=1; j=e+1; continue } } + j=skipval(s,j); if(!j) exit 2 } + if (!oclosed) exit 2 + if (ttype=="text") { if (!hastext || !textisstr) exit 1; printf "%s", ttext; exit 0 } + i=j } + exit 2 # ran off the end without a closing ] +} +' + +# Feed $payload UNCHANGED to the scan and return awk status untouched. No pre-processing, +# no jq, no collapsing of statuses. The program accumulates records and works in END, +# restoring the newline awk stripped, and prints with `printf "%s"` so no trailing newline +# is appended — the caller reads it through command substitution, which strips trailing +# newlines, harmless only because a JSON string cannot contain a raw newline. +locate_result() { printf '%s' "$payload" | awk "$LOCATE_AWK"; } + +# --- THE MATCHER, over the ESCAPED bytes the locator produced ------------------------- +# strip_ws is BOUNDED at 64 units: real JSON whitespace runs are 0-3 bytes, and an +# unbounded shell loop over externally supplied text is work an arbitrarily large result +# can dictate. Past the bound the leading whitespace remains, so the prefix match fails +# and the block is `unrecognized` — counted and disclosed, the fail-open direction. +strip_ws() { # strips leading literal spaces and the TWO-BYTE escapes \n \t \r + _v=$1; _i=0 + while [ "$_i" -lt 64 ]; do + case "$_v" in + ' '*) _v=${_v# } ;; + '\n'*|'\t'*|'\r'*) _v=${_v#??} ;; + *) break ;; + esac + _i=$((_i + 1)) + done + printf '%s' "$_v" +} + +# Is the value token `true`/`false` and nothing glued to it? `true*` alone would accept +# `truely`, which is a false SUCCESS — the dangerous direction. +_token_ends() { # $1 = the bytes after the literal + case "$(strip_ws "$1")" in '' | ,* | '}'*) return 0 ;; esac + return 1 +} + +classify_block() { + b=$1 + # no-result: blank on the located bytes. One sed pass, so a huge blank block costs + # linear work in sed rather than iterations of the bounded shell loop above. + # + # sed's STATUS is checked, because sed is load-bearing here: a missing or failing sed + # yields an empty substitution, which would read as blank and classify a genuine + # success envelope as `no-result` — fail-CLOSED, the one direction this design refuses + # for a result it can see. An unusable sed is uncertainty, so it goes to `unrecognized`. + # + # SCOPE: that guarantee is POST-ROUTING only. `field()` also needs sed on the jq-free + # path, so with jq absent AND sed absent `hook_event_name` and `tool_name` come back + # empty, the payload routes nowhere, and classification is never reached — nothing is + # counted and nothing is disclosed. Read as unconditional, the sentence above would + # promise a disclosure the hook has no path to make. + if _stripped=$(printf '%s' "$b" | sed 's/\\[ntr]//g; s/ //g'); then + case "$_stripped" in '') printf no-result; return ;; esac + else + printf unrecognized; return + fi + # backgrounded: the anchor at start of text, with `" is still running after ` + # occurring BEFORE any newline — ${b%%\\n*} is the text up to the first \n escape. + # + # TWO guards, and the second exists because the first was not enough. `%%` removes the + # LONGEST matching suffix, which bash 3.2 — macOS /bin/sh, what this hook runs under — + # evaluates by trying successively longer suffixes, at a cost quadratic in the distance + # to the first `\n`. Measured: 5.4 s at 150 KB, 21.6 s at 300 KB, in ONE synchronous + # invocation. + # + # 1. The literal anchor prefix. Not an optimization that changes what matches: + # ${b%%\n*} is a PREFIX of $b, so a block not starting with the anchor cannot + # produce a prefix that does. It skips the expansion for ordinary envelopes. + # 2. A BOUNDED head. Guard 1 alone still ran the full expansion on any block that + # does start with the anchor and then never completes the notice — 5.9 s at 150 KB + # (Gate-B pass 2 found this; the first timed fixture began with an envelope and + # took guard 1's cheap path, so it never saw the branch). Truncating first makes + # the expansion's input constant-size. A "does it contain \n" test does NOT fix + # it: with the newline near the END the expansion still scans to it, 5.5 s. + # + # `printf '%.Ns'` is POSIX string precision and works in both sh and dash (verified). + # THE LIMIT THIS BUYS, stated rather than left to be discovered: the segment must fall + # within the first 4096 characters, so a notice whose quoted tool name is ~4070+ chars + # long is classified `unrecognized` instead of `backgrounded` — it COUNTS rather than + # being discarded, which is the wrong direction. 4096 is ~200x the longest real tool + # name, and this is the same kind of bounded backstop as the 64-unit whitespace bound + # and the 200-frame depth cap, failing the same way. + case "$b" in + 'MCP tool \"'*) + _bg_head=$(printf '%.4096s' "$b") + _bg_head=${_bg_head%%\\n*} + case "$_bg_head" in + 'MCP tool \"'*'\" is still running after '*) printf backgrounded; return ;; + esac ;; + esac + # envelope polarity, immediately-first, tolerating encoded whitespace at each of + # the three grammar points: after `{`, after the key, after the colon. + case "$b" in '{'*) ;; *) printf unrecognized; return ;; esac + p=$(strip_ws "${b#\{}") + case "$p" in '\"success\"'*) ;; *) printf unrecognized; return ;; esac + p=$(strip_ws "${p#'\"success\"'}") + case "$p" in ':'*) ;; *) printf unrecognized; return ;; esac + v=$(strip_ws "${p#:}") + case "$v" in + true*) if _token_ends "${v#true}"; then printf success; return; fi ;; + false*) if _token_ends "${v#false}"; then printf failure; return; fi ;; + esac + printf unrecognized +} + +# `[ "$rc" = 0 ] ||` rather than `[ "$rc" = 2 ] &&`: a missing or failing awk exits 127, +# and every status that is not "located" or "nothing there" must reach the fail-open class +# rather than fall through to a matcher holding an empty string. +classify() { + blk=$(locate_result); rc=$? + [ "$rc" = 1 ] && { printf 'no-result'; return 0; } + [ "$rc" = 0 ] || { printf 'unrecognized'; return 0; } + classify_block "$blk" +} + +# --- OUTPUT BUFFER ---------------------------------------------------------------- +# Spec §6: an invocation that owes TWO messages composes them into ONE document. That is +# impossible while each branch writes as it decides, so branches call `note` and the +# single write happens in `flush_notes`, immediately before the final exit. `emit` is +# called from `flush_notes` and nowhere else. +# +# A7 — SEPARATORS AND ENCODING. `additionalContext` bodies join with " — " (space, em +# dash, space); `systemMessage` bodies with a single space. NO NEWLINE anywhere: the +# jq-free emitter escapes only backslash and quote, so a literal newline would produce an +# invalid JSON document. +notes_ctx='' +notes_msg='' +have_notes=0 +note() { # $1 = additionalContext body, $2 = systemMessage body + if [ "$have_notes" -eq 0 ]; then + notes_ctx="$1"; notes_msg="$2" + else + notes_ctx="$notes_ctx — $1"; notes_msg="$notes_msg $2" + fi + have_notes=1 +} + +# Same buffer, opposite end. Spec §6 fixes the composed order as **disclosure first, then +# the per-occurrence message**, and the per-occurrence message is buffered during routing +# while the disclosure is only decided in the flush — so the flush has to prepend. It used +# to call `note`, which appended, silently inverting a settled order; the golden froze the +# inversion, so nothing caught it until a reviewer read the spec against the test. +note_front() { # $1 = additionalContext body, $2 = systemMessage body + if [ "$have_notes" -eq 0 ]; then + notes_ctx="$1"; notes_msg="$2" + else + notes_ctx="$1 — $notes_ctx"; notes_msg="$2 $notes_msg" + fi + have_notes=1 +} + +# --- THE DIAGNOSTIC INTERFACE ------------------------------------------------------- +# `note_unverified` sets a flag and NOTHING else: the decision is here, the delivery and +# the state are in the flush, because only the flush knows whether anything was written. +mark_noted=0 +mark_bgadv=0 +want_unverified=0 +note_unverified() { want_unverified=1; } +note_discarded() { # $1 = the class whose pass is being discarded + case "$1" in + failure) note "$FAILURE_CTX" "$FAILURE_MSG" ;; + no-result) note "$NORESULT_CTX" "$NORESULT_MSG" ;; + backgrounded) + if [ -f "$bgadv_file" ]; then + note "$BG_SHORT_CTX" "$BG_SHORT_MSG" + else + note "$BG_LONG_CTX" "$BG_LONG_MSG"; mark_bgadv=1 + fi + ;; + esac +} + +# One-shot markers are applied HERE, not at the branch that asked for them, because only +# the flush knows whether anything was actually written. `emit` status 0 alone means +# shown; 1 (suppressed) and 2 (write failed) both leave the one-shot unspent. +# +# The A5 marker table, in code. `Earlier: ` says whose call the statement describes, not +# where the debt came from: a carried disclosure must not read as a statement about the +# current call, and when the current call is ITSELF unrecognized the statement IS about +# it, so no prefix — which is why `want_unverified` takes precedence over the pending +# check. Every marker write is best-effort: a failure proceeds and exits 0, with the debt +# retained (duplicate beats loss) rather than dropped. +flush_notes() { + # shown + owed cannot both stand: drop the debt, best-effort, and disclose nothing. + if [ -f "$unver_file" ] && [ -f "$pend_file" ]; then + rm -f "$pend_file" 2>/dev/null || true + fi + _disclose=0 + _prefix='' + if [ "$want_unverified" -eq 1 ]; then + [ -f "$unver_file" ] || _disclose=1 + elif [ ! -f "$unver_file" ] && [ -f "$pend_file" ]; then + _disclose=1 + _prefix='Earlier: ' + fi + if [ "$_disclose" -eq 1 ]; then + note_front "$_prefix$UNVERIFIED_CTX" "$_prefix$UNVERIFIED_MSG" + fi + [ "$have_notes" -eq 1 ] || return 0 + emit "$notes_ctx" "$notes_msg" + _flushed=$? + if [ "$_flushed" -eq 0 ]; then + mkdir -p "$state_dir" 2>/dev/null + if [ "$mark_noted" -eq 1 ]; then + # `printf '%s' '' > f`, NEVER `: > f`. `:` is a POSIX SPECIAL BUILTIN, so a + # redirection failure on one makes the shell EXIT — ignoring this `{ … } + # 2>/dev/null || true` and even an enclosing `if`. With a directory at the target + # the `:` form exits 2 under dash, and Ubuntu's /bin/sh IS dash. Invariant 1 says + # the hook always exits 0; this is the form that keeps it. + { printf '%s' '' > "$noted_file"; } 2>/dev/null || true + fi + if [ "$mark_bgadv" -eq 1 ]; then + { printf '%s' '' > "$bgadv_file"; } 2>/dev/null || true + fi + if [ "$_disclose" -eq 1 ]; then + if { printf '%s' '' > "$unver_file"; } 2>/dev/null; then + rm -f "$pend_file" 2>/dev/null || true + else + # The debt outlives the marker it could not write. + { printf '%s' '' > "$pend_file"; } 2>/dev/null || true + fi + fi + elif [ "$_disclose" -eq 1 ] && [ ! -f "$pend_file" ]; then + mkdir -p "$state_dir" 2>/dev/null + { printf '%s' '' > "$pend_file"; } 2>/dev/null || true fi return 0 } @@ -357,10 +799,26 @@ is_docs_only() { return 0 } +# `routed` gates the flush below. Spec §3.3: a payload the hook cannot ROUTE produces no +# output and touches no state — but `flush_notes` used to run unconditionally, so a pending +# disclosure was emitted (and cleared) on an event no branch here recognized. That is also +# how an untrusted event name reached the fallback emitter. +routed=0 case "$event" in PostToolUse) + routed=1 case "$tool" in "$review_tool") + # CLASSIFY BEFORE TOUCHING ANY PASS STATE. Three classes write none of it and + # say why; `success` and `unrecognized` behave exactly as before, the second + # adding the disclosure that the count was taken without inspection. There is no + # `exit 0` in the discarded branch — the message just buffered has to reach + # `flush_notes` at the bottom. + cls=$(classify) + case "$cls" in + failure | no-result | backgrounded) note_discarded "$cls" ;; + *) + [ "$cls" = unrecognized ] && note_unverified mkdir -p "$state_dir" 2>/dev/null h=$(tree_hash) prev=$(cat "$state_file" 2>/dev/null || echo '') @@ -381,18 +839,33 @@ case "$event" in fi { printf '%s' "$h" > "$state_file"; } 2>/dev/null || true bump_count "$count_file" + ;; + esac + ;; + "$exec_tool") + cls=$(classify) + case "$cls" in + failure | no-result | backgrounded) note_discarded "$cls" ;; + *) + [ "$cls" = unrecognized ] && note_unverified + mkdir -p "$state_dir" 2>/dev/null + bump_count "$countA_file" + ;; + esac ;; - "$exec_tool") mkdir -p "$state_dir" 2>/dev/null; bump_count "$countA_file" ;; mcp__codex__*) # FINDING F: a Codex server is connected, but under tool names the gates can't # attribute. Left silent, this is the worst failure mode the hook has: reviews # run, counters stay 0, and the STOP fires on every commit forever — which # trains the user to ignore the hook. Say it once (the marker), not per call. + # + # Target model: Claude Sonnet 4.5 / Opus 4.1 via Claude Code — same family and + # same prompting-guidance check as the five constants above. DOUBLE-quoted, + # unlike those, because it interpolates the three tool names, which is also why + # the `$` closing its matcher literal is escaped. if [ ! -f "$noted_file" ]; then - if emit "Codex tool '$tool' is not counted by the review gates. The gates count '$exec_tool' (Gate A, reviews TEXT) and '$review_tool' (Gate B, reviews a DIFF); your Codex server exposes a surface that cannot be attributed to one gate or the other, so passes made through it stay invisible and Gate B will keep reporting 'not run'. The fix is to install the pinned mcp-codex-dev server, which exposes both (/dev-workflow:workflow-init writes it into .mcp.json). Mapping the names in .context/codex-gate.tools ('execTool=' / 'reviewTool=') is only an option if your server genuinely has two tools that separate reviewing TEXT from reviewing a DIFF — pointing both gates at one general-purpose tool moves the counters without either gate meaning what it says, which is a false ✓ and worse than this note. Said once per workspace." "ℹ Codex tool '$tool' is not counted by the gates — see the note"; then - mkdir -p "$state_dir" 2>/dev/null - { : > "$noted_file"; } 2>/dev/null || true - fi + mark_noted=1 + note "Claude via Claude Code — gate hook. Codex tool '$tool' is not counted by the review gates. The gates count '$exec_tool' (Gate A, reviews TEXT) and '$review_tool' (Gate B, reviews a DIFF); your Codex server exposes a surface that cannot be attributed to one gate or the other. Passes made through it stay invisible and Gate B will keep reporting 'not run', because a gate that cannot attribute a call cannot credit it. Treat reviews run through this tool as uncounted and say so when you report gate status; do not read a satisfied count as covering them. Every remedy is a configuration change on the operator machine — installing a server, editing .mcp.json, writing a mapping file — so it is addressed to the operator in the note beside this one, and there is nothing here for you to apply. Said once per workspace, so treat a repeat as a marker problem rather than as new information." "ℹ Codex tool '$tool' is not counted by the gates. The gates count '$exec_tool' (Gate A, reviews TEXT) and '$review_tool' (Gate B, reviews a DIFF). Fix, in order of preference. First: install the pinned mcp-codex-dev server, which exposes both — /dev-workflow:workflow-init writes it into .mcp.json. Second, only if your server genuinely has two tools that separate reviewing TEXT from reviewing a DIFF: map the names in .context/codex-gate.tools ('execTool=' / 'reviewTool='). Pointing both gates at one general-purpose tool moves the counters without either gate meaning what it says, which is a false checkmark and worse than this note. Either way the tool name must start with mcp__codex__: this hook is invoked by a hooks.json matcher of ^(Bash|Skill|mcp__codex__.*)\$, so an out-of-namespace name is either never delivered to this hook at all — the mapping looks applied and does nothing — or, for the two reserved names Bash and Skill, is delivered and hijacks a lifecycle event. Both are refused. Register the server under the name codex to place its tools there." fi ;; Bash) @@ -426,12 +899,13 @@ case "$event" in esac ;; PreToolUse) + routed=1 case "$tool" in Bash) cmd=$(input_field command) if is_commit "$cmd"; then if is_wip_commit "$cmd"; then - emit "WIP commit — cycle-internal, per $policy: this exists so mcp__codex__review has a non-empty range to read (baseSha = this commit's parent). Gate B is not evaluated here and your pass counters are preserved. Run the review against this commit, then make the real commit when your final pass is clean." "ℹ WIP commit (Codex cycle preserved)" + note "WIP commit — cycle-internal, per $policy: this exists so mcp__codex__review has a non-empty range to read (baseSha = this commit's parent). Gate B is not evaluated here and your pass counters are preserved. Run the review against this commit, then make the real commit when your final pass is clean." "ℹ WIP commit (Codex cycle preserved)" else # Docs-only commits (spec/plan .md files) carry no code diff, # so Gate B (mcp__codex__review reviews a code diff) cannot apply — emit a @@ -441,7 +915,7 @@ case "$event" in has_all_flag "$cmd" && files=$(printf '%s\n%s\n' "$files" "$(git -C "$repo_root" diff --name-only 2>/dev/null)") files=$(printf '%s\n' "$files" | sed '/^$/d') if is_docs_only "$files"; then - emit "Docs-only commit — no code is staged, so Codex Gate B (mcp__codex__review reviews a code diff) does not apply here. If this commit includes a spec or plan, confirm it went through Gate A (mcp__codex__exec) instead." "ℹ Codex Gate B N/A (docs-only commit)" + note "Docs-only commit — no code is staged, so Codex Gate B (mcp__codex__review reviews a code diff) does not apply here. If this commit includes a spec or plan, confirm it went through Gate A (mcp__codex__exec) instead." "ℹ Codex Gate B N/A (docs-only commit)" else passes=$(read_count "$count_file") fresh=$(read_count "$fresh_file") @@ -456,7 +930,7 @@ case "$event" in # preserves the older, now-stale fingerprint, which reaches the STALE # branch below, not this one. The message names the absent FINGERPRINT, # not an absent review. - emit "STOP — Codex Gate B not satisfied: no fingerprint is recorded for this cycle — either no mcp__codex__review has run, or the last one's fingerprint could not be written or read back. Per $policy you MUST reach a minimum of $floor passes per cycle. Run Gate B (mcp__codex__review) now; if this repeats, check that .context/ and the state file inside it are readable and writable, and if the file exists but is unreadable or empty, delete it and run a fresh pass." "⚠ Codex Gate B: no recorded review" + note "STOP — Codex Gate B not satisfied: no fingerprint is recorded for this cycle — either no mcp__codex__review has run, or the last one's fingerprint could not be written or read back. Per $policy you MUST reach a minimum of $floor passes per cycle. Run Gate B (mcp__codex__review) now; if this repeats, check that .context/ and the state file inside it are readable and writable, and if the file exists but is unreadable or empty, delete it and run a fresh pass." "⚠ Codex Gate B: no recorded review" # `unavailable` on EITHER side is never a match: an uncomputable fingerprint # must read as unverified, and two of them must not cancel out. elif [ "$current" = unavailable ] || [ "$reviewed" = unavailable ] || @@ -468,9 +942,9 @@ case "$event" in # that the tree changed — under a repeated computation failure nothing # changed, and under a failed state write the content may be exactly what # was reviewed. - emit "STOP — Codex Gate B not satisfied: the hook cannot confirm that the content you are about to commit is the content mcp__codex__review last saw ($passes recorded pass(es) this cycle). Usually that means the working tree or the index changed since the review. It can also mean you only staged already-reviewed content — the bytes are fine, but the hook cannot tell staging from editing; that this hook was upgraded and the recorded fingerprint uses the older format (see CHANGELOG); or that the fresh fingerprint could not be computed or could not be stored. Run Gate B (mcp__codex__review) now — one clean pass is the complete remedy for the staging and post-upgrade cases too. If a fresh pass leaves this unchanged with nothing edited in between, the fault is in the machinery rather than the code: check that .context/ is writable, that TMPDIR is writable, that a checksum tool (shasum, sha1sum or cksum) runs, that git status works, and that the disk is not full — then run one more pass to record a usable fingerprint. Per $policy you MUST re-review after every fix." "⚠ Codex Gate B not satisfied (cannot confirm review)" + note "STOP — Codex Gate B not satisfied: the hook cannot confirm that the content you are about to commit is the content mcp__codex__review last saw ($passes recorded pass(es) this cycle). Usually that means the working tree or the index changed since the review. It can also mean you only staged already-reviewed content — the bytes are fine, but the hook cannot tell staging from editing; that this hook was upgraded and the recorded fingerprint uses the older format (see CHANGELOG); or that the fresh fingerprint could not be computed or could not be stored. Run Gate B (mcp__codex__review) now — one clean pass is the complete remedy for the staging and post-upgrade cases too. If a fresh pass leaves this unchanged with nothing edited in between, the fault is in the machinery rather than the code: check that .context/ is writable, that TMPDIR is writable, that a checksum tool (shasum, sha1sum or cksum) runs, that git status works, and that the disk is not full — then run one more pass to record a usable fingerprint. Per $policy you MUST re-review after every fix." "⚠ Codex Gate B not satisfied (cannot confirm review)" elif [ "$passes" -lt "$floor" ]; then - emit "Codex Gate B floor NOT met: only $passes/$floor mcp__codex__review pass(es) since the last commit. Per $policy the review is a LOOP with a hard minimum of $floor passes — run more (the ONLY early exit is a pass that returned zero findings), or proceed only if $policy's skip rule applies to this change — if you cannot locate and check that rule, run the remaining passes." "⚠ Codex Gate B below floor ($passes/$floor)" + note "Codex Gate B floor NOT met: only $passes/$floor mcp__codex__review pass(es) since the last commit. Per $policy the review is a LOOP with a hard minimum of $floor passes — run more (the ONLY early exit is a pass that returned zero findings), or proceed only if $policy's skip rule applies to this change — if you cannot locate and check that rule, run the remaining passes." "⚠ Codex Gate B below floor ($passes/$floor)" else # Distinguish the two counts (Finding 9): the cycle total includes passes # made BEFORE later edits, so they carry a different fingerprint. @@ -479,7 +953,7 @@ case "$event" in # establish that Codex read these bytes (spec §7, and the review-range row # in todos.md). The stronger phrasing was here and was removed; do not # restore it as a clarity improvement. - emit "Codex Gate B: $passes/$floor pass(es) this cycle, of which $fresh cover the CURRENT content fingerprint (unchanged since that review). The floor counts the cycle; only the fresh pass(es) carry the same fingerprint as what you are committing. Per $policy, commit only if your final pass was clean — no new Blocker/Major." "✓ Codex Gate B satisfied ($passes/$floor cycle, $fresh on current fingerprint)" + note "Codex Gate B: $passes/$floor pass(es) this cycle, of which $fresh cover the CURRENT content fingerprint (unchanged since that review). The floor counts the cycle; only the fresh pass(es) carry the same fingerprint as what you are committing. Per $policy, commit only if your final pass was clean — no new Blocker/Major." "✓ Codex Gate B satisfied ($passes/$floor cycle, $fresh on current fingerprint)" fi fi fi @@ -490,13 +964,13 @@ case "$event" in superpowers:executing-plans | superpowers:subagent-driven-development) passesA=$(read_count "$countA_file") if [ "$passesA" -lt "$floor" ]; then - emit "Codex Gate A floor NOT met: only $passesA/$floor mcp__codex__exec pass(es) on this spec/plan. Per $policy Gate A is a LOOP with a hard minimum of $floor passes (start each instruction with the superpowers:brainstorming directive; the ONLY early exit is a pass that returned zero findings). Gate A has no content check behind it — this floor is the only thing keeping the spec review honest. Run more passes before executing." "⚠ Codex Gate A below floor ($passesA/$floor)" + note "Codex Gate A floor NOT met: only $passesA/$floor mcp__codex__exec pass(es) on this spec/plan. Per $policy Gate A is a LOOP with a hard minimum of $floor passes (start each instruction with the superpowers:brainstorming directive; the ONLY early exit is a pass that returned zero findings). Gate A has no content check behind it — this floor is the only thing keeping the spec review honest. Run more passes before executing." "⚠ Codex Gate A below floor ($passesA/$floor)" else # Deliberately weaker wording than Gate B (Finding 12): countA counts # mcp__codex__exec CALLS, bound to no artifact. Hashing the artifact would # be wrong — a spec is SUPPOSED to change between passes — so the hook # cannot verify what was reviewed, and must not imply that it did. - emit "Codex Gate A: $passesA/$floor mcp__codex__exec pass(es) on this spec/plan — floor met by COUNT ONLY. The hook counts calls; it cannot verify what was reviewed or that findings were addressed. Proceed only if your final pass was clean — no new Blocker/Major." "✓ Codex Gate A floor met ($passesA/$floor passes, count only)" + note "Codex Gate A: $passesA/$floor mcp__codex__exec pass(es) on this spec/plan — floor met by COUNT ONLY. The hook counts calls; it cannot verify what was reviewed or that findings were addressed. Proceed only if your final pass was clean — no new Blocker/Major." "✓ Codex Gate A floor met ($passesA/$floor passes, count only)" fi ;; esac @@ -505,4 +979,6 @@ case "$event" in ;; esac +# Unroutable payload: no output, no state, debt preserved for a later routable event. +[ "$routed" -eq 1 ] && flush_notes exit 0 diff --git a/plugins/dev-workflow/hooks/codex-gate.test.sh b/plugins/dev-workflow/hooks/codex-gate.test.sh index e86b37f..9c22be6 100644 --- a/plugins/dev-workflow/hooks/codex-gate.test.sh +++ b/plugins/dev-workflow/hooks/codex-gate.test.sh @@ -3,12 +3,39 @@ # (from an installed plugin: sh "$CLAUDE_PLUGIN_ROOT"/hooks/codex-gate.test.sh) set -u HOOK="$(cd "$(dirname "$0")" && pwd)/codex-gate.sh" +FIXTURES="$(cd "$(dirname "$0")" && pwd)/fixtures" + +# The shell the HOOK ITSELF runs under, resolved to an absolute path once. Every runner +# below goes through it, so `HOOK_SH=dash sh codex-gate.test.sh` really does execute all +# of the hook's code under dash — which is the only thing "green under sh and dash" can +# honestly mean. +# +# Before this was parameterized every runner hardcoded `sh` or `/bin/sh`, so running the +# FILE under dash exercised the HARNESS under dash and the hook under whatever `/bin/sh` +# is — bash on macOS. Only the handful of explicit `dash "$HOOK"` rows below ever reached +# dash, and the release evidence generalized from them to all 437. Gate B caught it. +# +# Absolute, because the restricted-PATH runners replace PATH wholesale and a bare name +# would not resolve inside them. +HOOK_SH=${HOOK_SH:-sh} +HOOK_SH_BIN=$(command -v "$HOOK_SH") \ + || { printf 'FATAL: HOOK_SH=%s not found on PATH\n' "$HOOK_SH"; exit 1; } +case "$HOOK_SH_BIN" in + /*) ;; + *) printf 'FATAL: HOOK_SH=%s did not resolve to an absolute path (%s)\n' "$HOOK_SH" "$HOOK_SH_BIN"; exit 1 ;; +esac +printf '# hook under test runs with: %s\n' "$HOOK_SH_BIN" fails=0 pass() { printf 'ok - %s\n' "$1"; } fail() { printf 'FAIL - %s\n' "$1"; fails=$((fails + 1)); } work=$(mktemp -d) -trap 'rm -rf "$work"' EXIT +# Restricted-PATH and shim directories live OUTSIDE the test repo. Under $work they +# would be untracked worktree content, so creating one would move the hook's own +# worktree tree id (invariant 3) and every fingerprint assertion after it would drift +# for a reason no label mentions. +sandbox=$(mktemp -d) +trap 'rm -rf "$work" "$sandbox"' EXIT cd "$work" || exit 1 git init -q git config user.email t@t; git config user.name t @@ -24,18 +51,118 @@ countA=".context/codex-gate.passCountA" floorf=".context/codex-gate.floor" toolsf=".context/codex-gate.tools" notedf=".context/codex-gate.toolNote" +offf=".context/codex-gate.off" +# The three diagnostic markers. They are written from 0.8.0 onward; naming them here +# lets `reset_all` clear them from the first task, so no section inherits one. +bgadvf=".context/codex-gate.bgAdvice" +unverf=".context/codex-gate.unverified" +pendf=".context/codex-gate.unverifiedPending" # A HEAD commit must exist so `git diff HEAD` (the tree-hash input) is meaningful. printf 'v1\n' > app.ts git add app.ts >/dev/null 2>&1 git commit -qm init >/dev/null 2>&1 -run() { printf '%s' "$1" | sh "$HOOK"; } -rev() { run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}' >/dev/null; } +# --- payload builders ------------------------------------------------------------- +# `printf` and `cat` only, never `jq`: the suite has to run where jq does not, and a +# jq-built payload would make the driver the thing under test instead of the hook. +payload() { # $1 = tool name, $2 = a tool_response ARRAY + printf '{"hook_event_name":"PostToolUse","tool_name":"%s","tool_input":{},"tool_response":%s}' "$1" "$2" +} +resp() { printf '[{"type":"text","text":"%s"}]' "$1"; } # $1 = ESCAPED text bytes +resp_from() { cat "$FIXTURES/$1.response.json"; } +resp_success() { resp_from shape0-success; } +unrec() { resp 'not an envelope this hook knows'; } +# Retarget a whole captured fixture to another tool name. `sed` with no /g replaces the +# FIRST match on the line, and every fixture carries its top-level "tool_name" before +# "tool_response", so no response byte is touched. +payload_from() { sed "s|\"tool_name\":\"[^\"]*\"|\"tool_name\":\"$2\"|" "$FIXTURES/$1.json"; } + +# --- runners. Named for what they are: a SILENT runner behind a message assertion +# --- makes that assertion vacuous, so the capturing ones say so. +run() { printf '%s' "$1" | "$HOOK_SH_BIN" "$HOOK"; } # capturing +rev() { run "$(payload mcp__codex__review "$(resp_success)")" >/dev/null; } # silent +revout() { run "$(payload mcp__codex__review "$(resp_success)")"; } # capturing +execp() { run "$(payload mcp__codex__exec "$(resp_success)")" >/dev/null; } # silent +# The ONE result-less gate payload the suite still builds, kept so the `no-result` +# path has an input. Everything else that should count carries a real envelope. +rev_noresult() { run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}'; } +codextool() { run "$(payload "$1" "$(resp_success)")"; } # capturing +codextool_unrec() { run "$(payload "$1" "$(unrec)")"; } # capturing commitpre() { run '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m x"}}'; } commitpost() { run '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m x"}}' >/dev/null; } -reset_all() { rm -f "$state" "$count" "$fresh" "$countA" "$floorf" "$toolsf" "$notedf"; } -codextool() { run "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"$1\",\"tool_input\":{}}"; } + +# stdout closed / jq-free, in both combinations. Each RETURNS the hook's status rather +# than printing it: a `PATH=…` scalar cannot be expanded as a command prefix, so the +# redirection has to live inside a function. +run_closed() { printf '%s' "$1" | "$HOOK_SH_BIN" "$HOOK" >&- 2>/dev/null; } +nojq_run() { printf '%s' "$1" | PATH="$nojq" "$HOOK_SH_BIN" "$HOOK"; } +nojq_run_closed() { printf '%s' "$1" | PATH="$nojq" "$HOOK_SH_BIN" "$HOOK" >&- 2>/dev/null; } + +# --- restricted PATH builder. The command list is passed as positional parameters, not +# --- as an unquoted variable: `mk_path nojq $HOOK_CMDS` trips SC2086, and the repo's +# --- battery runs shellcheck --shell=sh over this file. +mk_path() { # $1 = directory name under $sandbox, $2.. = commands to link + _d="$sandbox/$1"; mkdir -p "$_d"; shift + for _t in "$@"; do + _p=$(command -v "$_t" 2>/dev/null) || continue + # `command -v` on a builtin returns a bare name, and symlinking that makes a dangling + # link that reports "present" to `command -v` inside the hook. + case "$_p" in /*) ;; *) continue ;; esac + ln -sf "$_p" "$_d/$_t" + done + printf '%s' "$_d" +} + +# EVERY external command the hook runs must be linked, not just the one being varied. +# tree_hash shells out to mktemp and cp; without them every jq-free Gate-B scenario +# computes `unavailable` and takes a different branch, so a matrix claiming to compare +# jq and jq-free would be comparing two different code paths. The oracle for that is +# asserted below, right after the PATH is built. +nojq=$(mk_path nojq cat grep sed head tr git mkdir rm cp mktemp awk shasum sha1sum cksum) + +# A jq-free PATH whose `sed` fails ONLY for the fallback emitter's escaping pass. Prints +# nothing when the real sed cannot be located, so the caller can skip its assertions. +mk_sedfail_path() { + _real=$(command -v sed 2>/dev/null) || return 0 + case "$_real" in /*) ;; *) return 0 ;; esac + _sd=$(mk_path sedfail cat grep head tr git mkdir rm cp mktemp awk shasum sha1sum cksum) + cat > "$_sd/sed" </dev/null +h_nojq=$(cat "$state" 2>/dev/null || echo '') +case "$h_nojq" in + '' | unavailable) fail "jq-free PATH records a usable fingerprint (got [$h_nojq])" ;; + *) pass "jq-free PATH records a usable fingerprint" ;; +esac +nojq_run "$(payload mcp__codex__review "$(resp_success)")" >/dev/null +[ "$(cat "$fresh" 2>/dev/null)" = 2 ] \ + && pass "jq-free fingerprint matches itself across two passes" \ + || fail "jq-free fingerprint matches itself across two passes" +reset_all # 1. SET on review + bump pass count rev @@ -222,7 +349,7 @@ reset_all; rev; rev; rev # 4. Gate A exec must NOT satisfy Gate B (separate state) reset_all -run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null +execp [ ! -f "$state" ] && pass "exec does not set Gate B" || fail "exec does not set Gate B" # 5. RESET on commit @@ -253,18 +380,18 @@ out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"com out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"echo digit commit"}}') [ -z "$out" ] && pass "embedded 'digit commit' -> silent (word boundary)" || fail "embedded 'digit commit' -> silent (word boundary)" -# 9. jq-absent fallback (best effort: build a PATH without jq) -nojq="$work/nojq"; mkdir -p "$nojq"; ok=1 -for t in git grep sed head cat mkdir rm touch awk shasum sha1sum cksum tr; do - p=$(command -v "$t" 2>/dev/null) && ln -s "$p" "$nojq/$t" 2>/dev/null || true -done -command -v git >/dev/null 2>&1 || ok=0 -if [ "$ok" = 1 ] && ! PATH="$nojq" command -v jq >/dev/null 2>&1; then +# 9. jq-absent fallback, on the SHARED $nojq built at the top of this file under +# $sandbox. This section used to REASSIGN $nojq to "$work/nojq" — untracked content +# inside the test repo, so creating it moved the hook's own worktree tree id +# (invariant 3) — and its command list omitted `cp`/`mktemp`, which tree_hash shells +# out to, so every later jq-free fingerprint row took the `unavailable` branch while +# its label claimed a jq/jq-free comparison. Section 0 asserts this PATH's contract. +if PATH="$nojq" command -v git >/dev/null 2>&1 && ! PATH="$nojq" command -v jq >/dev/null 2>&1; then reset_all - out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m x"}}' | PATH="$nojq" /bin/sh "$HOOK") + out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m x"}}' | PATH="$nojq" "$HOOK_SH_BIN" "$HOOK") printf '%s' "$out" | grep -q 'Gate B' && pass "jq-absent fallback works" || fail "jq-absent fallback works" # 9b. fallback must not be confused by a literal } inside the command - out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m }"}}' | PATH="$nojq" /bin/sh "$HOOK") + out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m }"}}' | PATH="$nojq" "$HOOK_SH_BIN" "$HOOK") printf '%s' "$out" | grep -q 'Gate B' && pass "jq-absent: brace in command still matches" || fail "jq-absent: brace in command still matches" else printf 'skip - jq-absent fallback (could not build jq-free PATH)\n' @@ -279,7 +406,7 @@ fi # 11. State-write failure must still exit 0 (special-builtin redirection regression) rm -rf .context; : > .context # make .context a FILE so the state dir cannot be created -printf '%s' '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}' | sh "$HOOK"; rc=$? +printf '%s' "$(payload mcp__codex__review "$(resp_success)")" | "$HOOK_SH_BIN" "$HOOK"; rc=$? [ "$rc" = 0 ] && pass "state-write failure still exits 0" || fail "state-write failure still exits 0 (got $rc)" rm -f .context; mkdir -p .context; : > .context/codex-gate.on # restore (incl. adoption) @@ -293,8 +420,13 @@ out=$(commitpre) out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:executing-plans"}}') [ -z "$out" ] && pass "off marker silences Gate A reminder" || fail "off marker silences Gate A reminder" -# 13. Full state machine keeps running while off (so re-enable is accurate) +# 13. Full state machine keeps running while off (same counting semantics as gate-on, +# not evidence of review) +# `reset_all` now clears the opt-out marker too, so it has to be re-set here — without +# this the two assertions below still pass, but with the gate ON, and the words "while +# off" in their labels would be false. reset_all +: > "$off" rev [ -f "$state" ] && pass "review still SETs state while off" || fail "review still SETs state while off" commitpost @@ -303,7 +435,7 @@ rm -f "$off" # re-enable reset_all rev; rev; rev out=$(commitpre) -printf '%s' "$out" | grep -q 'Gate B satisfied' && pass "re-enable sees accurate state" || fail "re-enable sees accurate state" +printf '%s' "$out" | grep -q 'Gate B satisfied' && pass "re-enable sees same counting semantics as gate-on, not evidence of review" || fail "re-enable sees same counting semantics as gate-on, not evidence of review" # 14. Below-floor reminder shows N/floor reset_all @@ -362,12 +494,12 @@ reset_all # 15. Gate A floor: exec bumps its own counter; plan-execution skill enforces + resets rm -f "$countA" -run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null +execp [ "$(cat "$countA" 2>/dev/null)" = 1 ] && pass "exec bumps Gate A count to 1" || fail "exec bumps Gate A count to 1" out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:executing-plans"}}') printf '%s' "$out" | grep -qE 'below floor|floor NOT met' && pass "1/3 exec -> Gate A below floor" || fail "1/3 exec -> Gate A below floor" -run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null -run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null +execp +execp out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:executing-plans"}}') printf '%s' "$out" | grep -q 'floor met' && pass "3/3 exec -> Gate A satisfied" || fail "3/3 exec -> Gate A satisfied" # FINDING 12: the Gate-A satisfied wording must NOT overstate — it counts calls only. @@ -377,7 +509,7 @@ run '{"hook_event_name":"PostToolUse","tool_name":"Skill","tool_input":{"skill": # 16. Every other Gate-A reset trigger zeroes a stale count for s in superpowers:brainstorming superpowers:writing-plans superpowers:subagent-driven-development; do - run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null + execp [ -f "$countA" ] || fail "setup: exec should create countA for $s" run "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"Skill\",\"tool_input\":{\"skill\":\"$s\"}}" >/dev/null [ ! -f "$countA" ] && pass "$s resets stale Gate A count" || fail "$s resets stale Gate A count" @@ -507,11 +639,11 @@ out=$(run '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"com # Silent is not enough: a non-adopted repo must be INERT. Writing state would litter an # unrelated project with a .context/ it never asked for. reset_all -rev; run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' >/dev/null +rev; execp [ ! -f "$state" ] && [ ! -f "$count" ] && [ ! -f "$countA" ] && pass "non-adopted repo writes no state" || fail "non-adopted repo writes no state" # ...even into a .context/ that does not exist yet (the dir itself must not appear) sub=$(mktemp -d); (cd "$sub" && git init -q && git config user.email t@t && git config user.name t && printf 'x\n' > a.ts && git add -A && git commit -qm i) >/dev/null 2>&1 -out=$(printf '%s' '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}' | (cd "$sub" && sh "$HOOK")) +out=$(printf '%s' "$(payload mcp__codex__review "$(resp_success)")" | (cd "$sub" && "$HOOK_SH_BIN" "$HOOK")) [ ! -d "$sub/.context" ] && pass "non-adopted repo: no .context/ directory created" || fail "non-adopted repo: no .context/ directory created" rm -rf "$sub" @@ -710,16 +842,16 @@ unborn=$(mktemp -d) mkdir -p .context; : > .context/codex-gate.on printf '1' > .context/codex-gate.floor # one pass is enough for this fixture printf 'x\n' > a.ts - R='{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{}}' - printf '%s' "$R" | sh "$HOOK" >/dev/null + R=$(payload mcp__codex__review "$(resp_success)") + printf '%s' "$R" | "$HOOK_SH_BIN" "$HOOK" >/dev/null h1=$(cat .context/codex-gate.gateB 2>/dev/null) - printf '%s' "$R" | sh "$HOOK" >/dev/null + printf '%s' "$R" | "$HOOK_SH_BIN" "$HOOK" >/dev/null h2=$(cat .context/codex-gate.gateB 2>/dev/null) [ -n "$h1" ] && [ "$h1" != unavailable ] && [ "$h1" = "$h2" ] || exit 1 # ...and the FIRST commit must actually be able to reach satisfied. Hashing and # self-matching is not enough: a consumer-side regression could still STOP every # first commit forever, which is the failure this fixture exists to catch. - out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit --allow-empty -m init"}}' | sh "$HOOK") + out=$(printf '%s' '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit --allow-empty -m init"}}' | "$HOOK_SH_BIN" "$HOOK") printf '%s' "$out" | grep -q 'Gate B satisfied' || exit 1 ) && pass "unborn repo hashes, self-matches, and can reach satisfied" \ || fail "unborn repo hashes, self-matches, and can reach satisfied" @@ -966,5 +1098,1067 @@ fi chmod 0644 "$state" 2>/dev/null reset_all; rev; rev; rev +# 33. emit REPORTS WHETHER IT ACTUALLY WROTE (spec §6). +# Marker-writing is conditional on "a complete hook JSON document was written", so a +# caller that dedupes a one-time note must be able to tell a real write from a failed +# one. Every assertion below is driven through the UNKNOWN-TOOL note, because it is +# the one branch that both emits and burns a one-shot marker: a review PostToolUse +# emits nothing, so closing stdout on it exercises no writer at all and both outcomes +# would hold before this change. +reset_all +run_closed "$(payload mcp__codex__codex "$(resp_success)")"; rc=$? +[ "$rc" = 0 ] && pass "hook exits 0 with stdout closed" || fail "hook exits 0 with stdout closed (got $rc)" +[ ! -f "$notedf" ] && pass "a failed write does not burn the one-shot" || fail "a failed write does not burn the one-shot" + +reset_all +nojq_run_closed "$(payload mcp__codex__codex "$(resp_success)")"; rc=$? +[ "$rc" = 0 ] && pass "jq-free: hook exits 0 with stdout closed" || fail "jq-free: hook exits 0 with stdout closed (got $rc)" +[ ! -f "$notedf" ] && pass "jq-free: failed write does not burn the one-shot" || fail "jq-free: failed write does not burn the one-shot" + +# The SELECTIVE sed shim. Removing sed outright breaks field()'s jq-free routing, so the +# hook would never reach emit and an encoder-failure test would pass for an unrelated +# reason. This one fails only a sed whose script carries the encoder's own substitution. +sedfail=$(mk_sedfail_path) +if [ -n "$sedfail" ] && + [ "$(printf 'x\n' | PATH="$sedfail" sed 's/x/y/' 2>/dev/null)" = y ] && + ! printf 'x\n' | PATH="$sedfail" sed 's/\\/\\\\/g; s/"/\\"/g' >/dev/null 2>&1 +then + # Both directions verified: ordinary substitutions still work, the encoder's fails. + reset_all + out=$(printf '%s' "$(payload mcp__codex__codex "$(resp_success)")" | PATH="$sedfail" "$HOOK_SH_BIN" "$HOOK" 2>/dev/null); rc=$? + [ "$rc" = 0 ] && pass "encoder failure exits 0" || fail "encoder failure exits 0 (got $rc)" + [ -z "$out" ] && pass "encoder failure prints nothing" || fail "encoder failure prints nothing (got [$out])" + [ ! -f "$notedf" ] && pass "encoder failure burns nothing" || fail "encoder failure burns nothing" +else + # The dependent assertions are skipped WITH the shim, not left to run against a + # working sed and report three green rows that tested nothing. + skip "selective sed shim unavailable — encoder-failure assertions not run" +fi +reset_all + +# 34. ONE INVOCATION WRITES ONE DOCUMENT (spec §6, A6/A7). +# Output is buffered and written once at the end, so every emitting branch must still +# produce EXACTLY one document — `= 1`, never `-le 1`, because `-le 1` passes a +# dropped message, which is the failure this whole section exists to catch. One +# document per line, so counting lines carrying the envelope key counts documents. +ndocs() { printf '%s' "$1" | grep -c 'hookSpecificOutput'; } +one_doc() { # $1 = label, $2 = captured output + n=$(ndocs "$2") + [ "$n" = 1 ] && pass "$1 emits exactly one document" || fail "$1 emits exactly one document (got $n)" +} + +git checkout -- app.ts >/dev/null 2>&1 +git reset -q >/dev/null 2>&1 +rm -f brand-new.ts NOTES.md; rm -rf docs +reset_all + +# 1/9 unknown Codex tool +one_doc "unknown-tool note" "$(codextool mcp__codex__codex)" +# 2/9 WIP commit +reset_all +one_doc "WIP commit" "$(run '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m \"wip: snapshot\""}}')" +# 3/9 docs-only commit +reset_all +mkdir -p docs && printf 'spec\n' > docs/plan.md && git add docs/plan.md >/dev/null 2>&1 +one_doc "docs-only commit" "$(commitpre)" +git reset -q >/dev/null 2>&1; rm -rf docs +# 4/9 no fingerprint recorded +reset_all +one_doc "no recorded fingerprint" "$(commitpre)" +# 5/9 stale fingerprint +reset_all; rev +printf 'churn\n' >> app.ts +one_doc "stale fingerprint" "$(commitpre)" +git checkout -- app.ts >/dev/null 2>&1 +# 6/9 below the Gate-B floor +reset_all; rev +one_doc "Gate B below floor" "$(commitpre)" +# 7/9 Gate B satisfied +reset_all; rev; rev; rev +one_doc "Gate B satisfied" "$(commitpre)" +# 8/9 Gate A below floor +reset_all +one_doc "Gate A below floor" "$(run '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:executing-plans"}}')" +# 9/9 Gate A floor met +reset_all; execp; execp; execp +one_doc "Gate A floor met" "$(run '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:executing-plans"}}')" + +# A SILENT event must stay silent. It has to be an event the hook ACTUALLY RECEIVES: +# hooks.json registers PostToolUse as ^(Bash|Skill|mcp__codex__.*)$, so a PostToolUse for +# Edit never arrives in production and a flush test built on it would prove nothing about +# a reachable invocation. A non-commit Bash PostToolUse is reachable and emits nothing. +reset_all +out=$(run '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"ls -la"}}') +[ -z "$out" ] && pass "a silent event emits nothing" || fail "a silent event emits nothing (got [$out])" + +# 35. INVARIANT 1 AT THE MARKER WRITE. `: > f` is a POSIX SPECIAL BUILTIN: a redirection +# failure on one makes the shell EXIT, ignoring the enclosing `{ … } 2>/dev/null || +# true` and even an `if`. With a DIRECTORY at the target the `:` form exits 2 under +# dash — and Ubuntu's /bin/sh IS dash, which is what CI runs. macOS sh does not expose +# it, which is why the existing regression test stood while the defect shipped. +reset_all +mkdir -p "$notedf" +codextool mcp__codex__codex >/dev/null 2>&1; rc=$? +[ "$rc" = 0 ] && pass "exits 0 with a directory at the marker path" || fail "exits 0 with a directory at the marker path (got $rc)" +if command -v dash >/dev/null 2>&1; then + printf '%s' "$(payload mcp__codex__codex "$(resp_success)")" | dash "$HOOK" >/dev/null 2>&1; rc=$? + [ "$rc" = 0 ] && pass "dash: exits 0 with a directory at the marker path" || fail "dash: exits 0 with a directory at the marker path (got $rc)" +else + skip "dash unavailable — the special-builtin assertion did not run" +fi +rmdir "$notedf" 2>/dev/null +reset_all + +# ===================================================================================== +# 36-41. GATE-PASS RESULT CLASSIFICATION (0.8.0). +# ===================================================================================== +# $nojq is the shared PATH from the top of this file; section 9 no longer reassigns it. +# Re-assert section 0's contract HERE anyway, immediately before the rows that depend on +# it: a PATH nobody re-checks at its point of use is how the earlier reassignment went +# unnoticed — every jq-free fingerprint row took the `unavailable` branch while its label +# claimed a jq/jq-free comparison, and each row still passed. +reset_all +nojq_run "$(payload mcp__codex__review "$(resp_success)")" >/dev/null +h_nojq=$(cat "$state" 2>/dev/null || echo '') +nojq_run "$(payload mcp__codex__review "$(resp_success)")" >/dev/null +case "$h_nojq" in + '' | unavailable) fail "jq-free PATH at point of use records a usable fingerprint (got [$h_nojq])" ;; + *) [ "$(cat "$fresh" 2>/dev/null)" = 2 ] \ + && pass "jq-free PATH at point of use records a usable, self-matching fingerprint" \ + || fail "jq-free PATH at point of use records a usable, self-matching fingerprint" ;; +esac +reset_all + +# --- classification helpers --------------------------------------------------------- +# The corpus is driven through the EXEC tool: it routes identically and skips tree_hash, +# which the classification rows do not test and which dominates the runtime. +cpay() { # $1 = the tool_response VALUE -> a routable gate payload + printf '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{},"tool_response":%s}' "$1" +} +ctext() { cpay "[{\"type\":\"text\",\"text\":\"$1\"}]"; } # $1 = ESCAPED text bytes + +# One invocation's class from THAT invocation's own effects: an exact message substring +# AND the pass-state effect. The substring alone would let a discarded class that still +# counted read as correctly discarded, which is the false ✓ direction. No cross-test +# global state: every call resets first. +class_of() { # $1 = a full hook payload + reset_all + _o=$(run "$1") + _n=$(( $(cat "$count" 2>/dev/null || printf 0) + $(cat "$countA" 2>/dev/null || printf 0) )) + case "$_o" in + *'this Codex call returned an envelope reporting failure'*) _k=failure ;; + *'carried no result text the hook could read'*) _k=no-result ;; + *'moved to the background at the auto-background threshold'* \ + | *'was backgrounded and its result never reached the hook'*) _k=backgrounded ;; + *'classified at least one gate call as countable'*) _k=unrecognized ;; + '') _k=success ;; + *) _k=other ;; + esac + case "$_k" in + success | unrecognized) [ "$_n" = 1 ] || _k="$_k+notcounted" ;; + failure | no-result | backgrounded) [ "$_n" = 0 ] || _k="$_k+counted" ;; + esac + printf '%s' "$_k" +} +cls_is() { # $1 = label, $2 = got, $3 = want + [ "$2" = "$3" ] && pass "class: $1" || fail "class: $1 (got [$2], want [$3])" +} +cls_text() { cls_is "$1" "$(class_of "$(ctext "$2")")" "$3"; } +cls_resp() { cls_is "$1" "$(class_of "$(cpay "$2")")" "$3"; } +cls_pay() { cls_is "$1" "$(class_of "$2")" "$3"; } + +# 36. THE CLASSIFICATION CORPUS, ported label-by-label from the verified drafts +# (.context/plan-drafts/verify.sh, 53 call sites / 56 assertions). Every row whose +# document is VALID JSON runs end-to-end through the hook; the rows whose OUTER +# document is malformed run at LOCATOR level in section 36b, because their +# routability — not their classification — differs between the jq and grep paths, +# which is not the classifier doing anything. + +# 36a/1-4. the four real captures, as whole payloads retargeted onto a gate tool. +for pair in shape0-success:success shape1-fast-fail:failure \ + shape2-executor-timeout:failure shape3-backgrounding-notice:backgrounded; do + f=${pair%%:*}; want=${pair##*:} + cls_pay "fixture $f" "$(payload_from "$f" mcp__codex__exec)" "$want" +done +# ...and the review-tool capture, which must NOT reach the fail-open terminal class. +# Count and fingerprint alone cannot see that: assert the class. +cls_pay "fixture shape0-success-review" "$(payload_from shape0-success-review mcp__codex__exec)" success + +# 36a/5-11. polarity grammar (A3): the whitespace points, and the glued token. +cls_text "compact" '{\"success\":true}' success +cls_text "tab around colon" '{\t\"success\"\t:\ttrue}' success +cls_text "CRLF after brace" '{\r\n\"success\": false}' failure +cls_text "space before colon" '{ \"success\" : true}' success +cls_text "reordered envelope" '{\"status\": \"error\", \"success\": false}' unrecognized +cls_text "glued token truely" '{\"success\": truely}' unrecognized +ws70=$(i=0; while [ $i -lt 70 ]; do printf ' '; i=$((i + 1)); done) +cls_text "whitespace past 64 after the brace" "{$ws70\\\"success\\\": true}" unrecognized +# P9-3: one bound row per remaining strip_ws site, not only the one after `{`. +cls_text "whitespace past 64 after the key" "{\\\"success\\\"$ws70: true}" unrecognized +cls_text "whitespace past 64 after the colon" "{\\\"success\\\":${ws70}true}" unrecognized +cls_text "whitespace past 64 after the value" "{\\\"success\\\": true$ws70}" unrecognized + +# 36a/12-13. collisions: a summary quoting the marker literals. The FAILURE direction is +# where a mistake produces the false ✓, so both directions are pinned — synthetically +# here, and through the shipped collision fixtures immediately after. +cls_text "success quotes both" '{\"success\": true, \"summary\": \"x \\\"success\\\": false\"}' success +cls_text "failure quotes true" '{\"success\": false, \"summary\": \"x \\\"success\\\": true\"}' failure +cls_pay "fixture collision-success-quotes-both" \ + "$(payload_from collision-success-quotes-both mcp__codex__exec)" success +cls_pay "fixture collision-failure-quotes-true" \ + "$(payload_from collision-failure-quotes-true mcp__codex__exec)" failure + +# 36a/14-19. the backgrounding anchor (A4) and its four near-misses. +cls_text "notice, exec" 'MCP tool \"codex/exec\" is still running after 120s (task abc)' backgrounded +cls_text "notice, mapped" 'MCP tool \"other/thing\" is still running after 5m' backgrounded +cls_text "notice quoted in an envelope" \ + '{\"success\": false, \"summary\": \"MCP tool \\\"codex/exec\\\" is still running after 120s\"}' failure +cls_text "prefix without the segment" 'MCP tool \"codex/exec\" finished' unrecognized +cls_text "segment after a newline" 'MCP tool \"codex/exec\"\n is still running after 120s' unrecognized +cls_text "phrase later in the text" 'The gate is still running after the fix' unrecognized + +# 36a/20-28. every no-result shape (spec §3.3) — none of them may count. +cls_pay "absent tool_response" \ + '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{}}' no-result +cls_resp "null tool_response" 'null' no-result +cls_resp "empty array" '[]' no-result +cls_resp "non-array container" '{"type":"text","text":"x"}' no-result +cls_resp "no text-type element" '[{"type":"image","data":"x"}]' no-result +cls_resp "text not a string" '[{"type":"text","text":123}]' no-result +cls_text "empty text" '' no-result +cls_text "blank escaped ws" ' \n\t ' no-result +cls_text "unicode-escaped space" '\u0020' unrecognized + +# 36a/29-31. locating: decoys in both directions, and block selection. +cls_pay "tool_input decoy quotes the key BEFORE the real field" \ + '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"see \"tool_response\" docs"},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' \ + success +cls_resp "non-object element skipped" '["x",{"type":"text","text":"{\"success\": true}"}]' success +# BLOCK SELECTION: an `element [0]` implementation passes every no-result and capture row +# while violating spec §3.1's settled first-`text`-element rule. This row is what fails. +cls_resp "non-text block skipped, classified from the text block" \ + '[{"type":"image","data":"x"},{"type":"text","text":"{\"success\": false}"}]' failure + +# 36a/32-33. duplicate depth-1 keys — last-wins would classify an ambiguous payload. +cls_pay "duplicate depth-1 key" \ + '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_response":[{"type":"text","text":"{\"success\": true}"}],"tool_response":[{"type":"text","text":"{\"success\": false}"}]}' \ + unrecognized +cls_pay "duplicate depth-1 key, reversed" \ + '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_response":[{"type":"text","text":"{\"success\": false}"}],"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' \ + unrecognized + +# 36a/34-37. duplicate classification-relevant members. The PRECEDING-element rows pin a +# refusal stricter than the first-text-element rule requires: without them the generic +# duplicate rows are all satisfied by selected-element duplicates and the stricter +# behaviour could regress with every named check still green. P9-5 adds the `text` +# counterpart to the `type` one the drafts carried. +cls_resp "duplicate text member" \ + '[{"type":"text","text":"{\"success\": false}","text":"{\"success\": true}"}]' unrecognized +cls_resp "duplicate type member" \ + '[{"type":"image","type":"text","text":"{\"success\": true}"}]' unrecognized +cls_resp "duplicate type in a PRECEDING non-text element" \ + '[{"type":"image","type":"image"},{"type":"text","text":"{\"success\": true}"}]' unrecognized +cls_resp "duplicate text in a PRECEDING non-text element" \ + '[{"type":"image","text":"a","text":"b"},{"type":"text","text":"{\"success\": true}"}]' unrecognized + +# 36a/38-39. ordinary nesting is unaffected, and a later mention of the key does not +# return a byte-position heuristic to life. +shallow=$(awk 'BEGIN{s="";for(i=0;i<8;i++)s=s "[";for(i=0;i<8;i++)s=s "]";print s}') +cls_pay "ordinary nesting is unaffected" \ + "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"mcp__codex__exec\",\"junk\":$shallow,\"tool_response\":[{\"type\":\"text\",\"text\":\"{\\\"success\\\": true}\"}]}" \ + success +cls_resp "response quotes the key AFTER the real field" \ + '[{"type":"text","text":"{\"success\": true} plus a later mention of \\\"tool_response\\\""}]' success + +# 36a/40. the key SPELLED with Unicode escapes, six bytes each. The locator must SUCCEED +# and the matcher must refuse — an earlier draft supplied raw quotes, which made the outer +# payload malformed and reached `unrecognized` from the locator instead. +cls_text "unicode-escaped marker key" '{\u0022success\u0022: true}' unrecognized + +# 36a/40b. P9-6, CHARACTERIZED rather than fixed — this row exists so the boundary is a +# tested fact instead of an unexamined assumption. The `type` VALUE is compared as raw +# bytes, so a Unicode-escaped spelling of `text` — semantically `text`, and valid JSON a +# conforming serializer may emit — is not recognized as a text element. With no other +# element the locator reports "nothing there" and the class is `no-result`: fail-CLOSED, +# so a real result is discarded rather than miscounted, which is the safe direction but +# is still a wrong verdict on a legal payload. Spec 3.1's "first text element" is a +# SEMANTIC contract while this is a byte comparison; the gap is stated in the spec and +# carried in todos.md with a trigger. If the comparison is ever made semantic, this is +# the row that must change — deliberately, not by accident. +cls_pay "P9-6 escaped type value reads as no-result" \ + "$(payload mcp__codex__exec '[{"type":"\u0074ext","text":"{\"success\": true}"}]')" no-result + +# 36a/41-42. the length ceiling, and a pretty-printed payload. +big=$(awk 'BEGIN{s="";while(length(s)<1100000)s=s "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";print s}') +cls_pay "payload past the ceiling" \ + "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"mcp__codex__exec\",\"pad\":\"$big\",\"tool_response\":[{\"type\":\"text\",\"text\":\"{\\\"success\\\": true}\"}]}" \ + unrecognized +unset big +cls_pay "pretty-printed payload" '{ + "hook_event_name": "PostToolUse", + "tool_name": "mcp__codex__exec", + "tool_response": [ + { "type": "text", "text": "{\"success\": true}" } + ] +}' success + +# 36b. LOCATOR-LEVEL ROWS. Every remaining draft row has a MALFORMED OUTER document, so +# whether the hook routes it at all depends on jq (jq refuses, the grep fallback +# reads a tool name anyway) — two different behaviours in two environments, neither +# of them the classifier deciding anything. Their claim is about the locator, so they +# are asserted at that level, against the hook's OWN embedded program rather than a +# copy that could drift. +extract_locate_awk() { sed -n "/^LOCATE_AWK='\$/,/^'\$/p" "$HOOK" | sed '1d;$d'; } +LOCATE_PROG=$(extract_locate_awk) +loc_verdict() { # $1 = a raw document -> located: | nothing | refused + _b=$(printf '%s' "$1" | awk "$LOCATE_PROG" 2>/dev/null); _r=$? + case "$_r" in + 0) printf 'located:%s' "$_b" ;; + 1) printf 'nothing' ;; + *) printf 'refused' ;; + esac +} +loc_is() { [ "$2" = "$3" ] && pass "locator: $1" || fail "locator: $1 (got [$2], want [$3])"; } +E_OK='{\"success\": true}' +# The extraction is self-verified in both directions before anything depends on it: a +# known-good document must locate its known bytes, and a known-refusable one must refuse. +# +# Every document below is built into `p` on its OWN LINE first. Writing +# `"$(loc_verdict "{\"a\":…}")"` inline does NOT survive the nesting — the outer quotes +# close early and the verdict is computed over the wrong bytes, which is exactly how the +# self-verification below first reported a working extraction as unusable, skipping all +# fifteen rows while every one of them would have passed. +EL="{\"type\":\"text\",\"text\":\"$E_OK\"}" +p="{\"tool_response\":[$EL]}" +sv1=$(loc_verdict "$p") +sv2=$(loc_verdict 'nonsense') +if [ -n "$LOCATE_PROG" ] && [ "$sv1" = "located:$E_OK" ] && [ "$sv2" = refused ]; then + loc_is "not an object" "$(loc_verdict '[1,2]')" refused + p="{\"tool_response\":[$EL" + loc_is "truncated array" "$(loc_verdict "$p")" refused + p="{\"tool_response\":[{\"type\":\"text\",\"text\":\"$E_OK\"" + loc_is "truncated element" "$(loc_verdict "$p")" refused + p="{\"junk\":[1,2},\"tool_response\":[$EL]}" + loc_is "mismatched container before the field" "$(loc_verdict "$p")" refused + p="{\"tool_response\":[{\"type\":\"text\",\"text\":\"$E_OK\",\"m\":[1}}]}" + loc_is "mismatched container inside the array" "$(loc_verdict "$p")" refused + p="{\"tool_response\":[$EL]}junk" + loc_is "trailing garbage after the object" "$(loc_verdict "$p")" refused + p="{\"tool_response\":[{\"type\":\"text\",,\"text\":\"$E_OK\"}]}" + loc_is "stray comma between members" "$(loc_verdict "$p")" refused + p="{\"tool_response\":[,$EL]}" + loc_is "malformation BEFORE the selected element refuses" "$(loc_verdict "$p")" refused + p="{\"junk\":tru,\"tool_response\":[$EL]}" + loc_is "bare primitive token" "$(loc_verdict "$p")" refused + p="{\"junk\":\"oops,\"tool_response\":[$EL]}" + loc_is "unterminated string before the field" "$(loc_verdict "$p")" refused + # The array is walked only as far as the selected element, so malformation AFTER it + # cannot change which bytes were located. Stated as behaviour rather than argued. + p="{\"tool_response\":[$EL,]}" + loc_is "trailing comma AFTER the selected element is not walked" "$(loc_verdict "$p")" "located:$E_OK" + # The DEPTH CAP at 201 openers, and the ordinary-nesting row that makes it safe to + # tighten: without the second, a cap set low enough to refuse real payloads still passes. + deep=$(awk 'BEGIN{s="";for(i=0;i<201;i++)s=s "[";print s}') + p="{\"junk\":$deep,\"tool_response\":[$EL]}" + loc_is "nesting past the depth cap" "$(loc_verdict "$p")" refused + unset deep + # WALKABLE-BUT-INVALID, pinned to TODAY's behaviour on purpose: the scan walks past the + # invalid part and locates the real block (status 0). If someone later tightens the + # locator into a validator, or re-widens a claim about validation, one of these moves. + p="{\"junk\":[1,],\"tool_response\":[$EL]}" + loc_is "walkable: balanced-but-invalid sibling array" "$(loc_verdict "$p")" "located:$E_OK" + p="{\"junk\":{\"a\" 1},\"tool_response\":[$EL]}" + loc_is "walkable: sibling member with no colon" "$(loc_verdict "$p")" "located:$E_OK" + p='{"junk":"\q","tool_response":['"$EL"']}' + loc_is "walkable: invalid string escape in a sibling" "$(loc_verdict "$p")" "located:$E_OK" +else + fail "locator program could not be extracted and self-verified (sv1=[$sv1] sv2=[$sv2]) — 15 locator-level rows not run" +fi + +# 36c. FIXTURE / SLICE PARITY — the permanent form of Task 1 Step 7, reading only tracked +# paths. A fixture edited without its slice leaves every classification row green +# while the README byte-exact claim about the duplicated representation is false. +# Both locator statuses must be 0 BEFORE the bytes are compared: two failed +# extractions produce two equal empty strings. +if [ -n "$LOCATE_PROG" ]; then + for f in shape0-success shape0-success-review shape1-fast-fail shape2-executor-timeout \ + shape3-backgrounding-notice collision-success-quotes-both collision-failure-quotes-true; do + a=$(awk "$LOCATE_PROG" < "$FIXTURES/$f.json" 2>/dev/null); ra=$? + b=$(printf '{"tool_response":%s}' "$(cat "$FIXTURES/$f.response.json")" | awk "$LOCATE_PROG" 2>/dev/null); rb=$? + if [ "$ra" = 0 ] && [ "$rb" = 0 ] && [ "$a" = "$b" ]; then + pass "$f: payload and response slice locate identical bytes" + else + fail "$f: payload and response slice locate identical bytes (rc $ra/$rb)" + fi + done +else + fail "locator program unavailable — 7 fixture/slice parity rows not run" +fi + +# 37. STATE EFFECTS. Three discarded classes x both gates x both tool-name sources x both +# emitters. `writes no gate-pass state from clean` and `preserves earned state` are +# different failures: byte preservation alone cannot see an implementation that +# recomputes and stores the CURRENT fingerprint over a seeded identical one, which is +# why the seed is a sentinel rather than a real hash. +MAPPING='execTool=mcp__codex__gateA +reviewTool=mcp__codex__gateB' +disc_payload() { # $1 = class, $2 = tool name + case "$1" in + failure) _tr=$(resp_from shape1-fast-fail) ;; + timeout) _tr=$(resp_from shape2-executor-timeout) ;; + no-result) _tr='null' ;; + backgrounded) _tr=$(resp_from shape3-backgrounding-notice) ;; + unrecognized) _tr=$(unrec) ;; + esac + printf '{"hook_event_name":"PostToolUse","tool_name":"%s","tool_input":{},"tool_response":%s}' "$2" "$_tr" +} +run_as() { # $1 = jq|nojq, $2 = payload (silent: these rows assert state, not messages) + case "$1" in + jq) run "$2" >/dev/null ;; + nojq) nojq_run "$2" >/dev/null ;; + esac +} +tool_for() { # $1 = source, $2 = gate + case "$1/$2" in + default/review) printf 'mcp__codex__review' ;; + default/exec) printf 'mcp__codex__exec' ;; + mapped/review) printf 'mcp__codex__gateB' ;; + mapped/exec) printf 'mcp__codex__gateA' ;; + esac +} +apply_mapping() { [ "$1" = mapped ] && printf '%s\n' "$MAPPING" > "$toolsf"; return 0; } +state_snapshot() { + printf '%s|%s|%s|%s' "$(cat "$state" 2>/dev/null)" "$(cat "$count" 2>/dev/null)" \ + "$(cat "$fresh" 2>/dev/null)" "$(cat "$countA" 2>/dev/null)" +} +# `timeout` is the CAPTURED shape2 executor-timeout envelope, driven through the same +# seeded-state matrix as the other discarded classes. The plan required a seeded-state +# TIMEOUT preservation row specifically; an earlier revision satisfied that with a +# classification-only row and the seeded matrix used the fast-failure fixture alone, so +# nothing proved the old hook advanced counters and fingerprint for the captured timeout +# payload under seeded state. Restored rather than re-dispositioned (Gate-B pass 4/6). +for k in failure timeout no-result backgrounded; do + for gate in review exec; do + for src in default mapped; do + for rn in jq nojq; do + tn=$(tool_for "$src" "$gate"); lbl="$k/$gate/$src/$rn" + reset_all; apply_mapping "$src" + run_as "$rn" "$(disc_payload "$k" "$tn")" + if [ ! -f "$state" ] && [ ! -f "$count" ] && [ ! -f "$fresh" ] && [ ! -f "$countA" ]; then + pass "$lbl writes no gate-pass state from clean" + else + fail "$lbl writes no gate-pass state from clean" + fi + reset_all; apply_mapping "$src" + printf 'SEEDED-FINGERPRINT' > "$state"; printf '7' > "$count" + printf '5' > "$fresh"; printf '4' > "$countA" + before=$(state_snapshot) + run_as "$rn" "$(disc_payload "$k" "$tn")" + [ "$(state_snapshot)" = "$before" ] \ + && pass "$lbl preserves passCount/freshCount/fingerprint/passCountA" \ + || fail "$lbl preserves passCount/freshCount/fingerprint/passCountA" + done + done + done +done +reset_all + +# 37b. `unrecognized` is the FAIL-OPEN class: it must behave exactly like `success` on +# state, over the same four combinations (P9-14), and additionally disclose. +for src in default mapped; do + for gate in review exec; do + for rn in jq nojq; do + tn=$(tool_for "$src" "$gate"); lbl="unrecognized/$gate/$src/$rn" + reset_all; apply_mapping "$src" + run_as "$rn" "$(disc_payload unrecognized "$tn")" + if [ "$gate" = review ]; then _got=$(cat "$count" 2>/dev/null || printf -) + else _got=$(cat "$countA" 2>/dev/null || printf -); fi + [ "$_got" = 1 ] && pass "$lbl counts" || fail "$lbl counts (got [$_got])" + [ -f "$unverf" ] && pass "$lbl records the disclosure marker" \ + || fail "$lbl records the disclosure marker" + done + done +done +reset_all + +# 37c. The two state families must not be conflated: a discarded class writes no +# DIAGNOSTIC marker either, except `backgrounded`, whose advice marker is by design. +for k in failure no-result; do + reset_all + run "$(disc_payload "$k" mcp__codex__review)" >/dev/null + if [ ! -f "$bgadvf" ] && [ ! -f "$unverf" ] && [ ! -f "$pendf" ]; then + pass "$k creates no diagnostic marker" + else + fail "$k creates no diagnostic marker" + fi +done +reset_all +run "$(disc_payload backgrounded mcp__codex__review)" >/dev/null +[ -f "$bgadvf" ] && pass "backgrounded creates its diagnostic marker" \ + || fail "backgrounded creates its diagnostic marker" +[ ! -f "$unverf" ] && [ ! -f "$pendf" ] \ + && pass "backgrounded creates no disclosure marker" \ + || fail "backgrounded creates no disclosure marker" + +# 37d. `success` and `unrecognized` have IDENTICAL counter and fingerprint effects by +# design, so only the marker and the message separate them. +reset_all +rev +[ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "success counts" || fail "success counts" +[ ! -f "$unverf" ] && [ ! -f "$pendf" ] \ + && pass "success creates no disclosure marker" || fail "success creates no disclosure marker" +reset_all + +# 38. THE A5 MARKER TABLE, row by row. Each needs a SURGICAL fault: replacing .context +# with a file removes the adoption marker so the hook exits before classifying, and +# chmod 500 .context breaks the counter writes while stdout still succeeds. What +# separates the operations: a DIRECTORY at a marker path fails `printf > f`. +unrec_rev() { run "$(disc_payload unrecognized mcp__codex__review)"; } + +# 38a. on: writes shown, owes nothing +reset_all +out=$(unrec_rev) +[ -f "$unverf" ] && [ ! -f "$pendf" ] && pass "on: writes shown, owes nothing" \ + || fail "on: writes shown, owes nothing" +printf '%s' "$out" | grep -q 'classified at least one gate call as countable' \ + && pass "on: the disclosure is delivered" || fail "on: the disclosure is delivered" + +# 38b. off: owes pending, no shown +reset_all; : > "$offf" +out=$(unrec_rev) +[ -z "$out" ] && pass "off: nothing is delivered" || fail "off: nothing is delivered" +[ -f "$pendf" ] && [ ! -f "$unverf" ] && pass "off: owes pending, no shown" \ + || fail "off: owes pending, no shown" +[ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "off: the unrecognized pass still counts" \ + || fail "off: the unrecognized pass still counts" +rm -f "$offf" + +# 38c. failed write: owes pending +reset_all +run_closed "$(disc_payload unrecognized mcp__codex__review)"; rc=$? +[ "$rc" = 0 ] && pass "failed write: exits 0" || fail "failed write: exits 0 (got $rc)" +[ -f "$pendf" ] && [ ! -f "$unverf" ] && pass "failed write: owes pending" \ + || fail "failed write: owes pending" + +# 38d. shown-write failure keeps the debt — the debt must not die with the marker. +reset_all +mkdir -p "$unverf" +unrec_rev >/dev/null 2>&1; rc=$? +[ "$rc" = 0 ] && pass "shown-write failure: exits 0" || fail "shown-write failure: exits 0 (got $rc)" +[ -f "$pendf" ] && pass "shown-write failure keeps the debt" || fail "shown-write failure keeps the debt" +rmdir "$unverf" 2>/dev/null +reset_all + +# 38e. a carried disclosure is PREFIXED, and clears on delivery. The setup must not itself +# emit, or it flushes the debt first: keep the gate off through setup and remove the +# marker immediately before the observed event. +reset_all; : > "$offf" +unrec_rev >/dev/null # earns the debt while suppressed +rm -f "$offf" +# `revout`, not `rev`: `rev` is the SILENT runner, and a message assertion behind it is +# vacuous — it reported an empty capture as a missing prefix. +out=$(revout) # a success pass: owes nothing of its own +printf '%s' "$out" | grep -q 'Earlier: Claude via Claude Code — gate hook' \ + && pass "pending flushes prefixed" || fail "pending flushes prefixed" +[ ! -f "$pendf" ] && [ -f "$unverf" ] && pass "flush clears pending" || fail "flush clears pending" + +# 38f. pending + a currently-unrecognized call: ONE message, NOT prefixed, and pending +# clears. The statement is about THIS call, so `note_unverified` takes precedence +# over the pending check — inverting that precedence is what this row fails on. +reset_all; : > "$offf" +unrec_rev >/dev/null +rm -f "$offf" +# The debt-earning call counted too, so the counters are zeroed WITHOUT touching the +# markers — otherwise `still counts` reads 2 and fails for a setup reason. +reset_gate_state +out=$(unrec_rev) +printf '%s' "$out" | grep -q 'Earlier: ' \ + && fail "pending+unrecognized: not prefixed" || pass "pending+unrecognized: not prefixed" +[ "$(printf '%s' "$out" | grep -c 'classified at least one gate call as countable')" = 1 ] \ + && pass "pending+unrecognized: exactly one disclosure" \ + || fail "pending+unrecognized: exactly one disclosure" +[ -f "$unverf" ] && [ ! -f "$pendf" ] && pass "pending+unrecognized: shown written, pending cleared" \ + || fail "pending+unrecognized: shown written, pending cleared" +[ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "pending+unrecognized: still counts" \ + || fail "pending+unrecognized: still counts" + +# 38g. shown + pending coexisting resolves to shown, silently. A REAL pending file is +# required: a directory there is not seen as pending at all. +# `revout`, NOT `rev`: `rev` redirects the hook's stdout to /dev/null, so a silence +# assertion behind it compares an always-empty capture and passes against a hook that +# shouts. Gate-B pass 5 mutation-tested exactly this row and it survived the bug. The +# suite names the two runners apart for this reason, and this is the second time the +# silent one has been used behind a message assertion. +reset_all +printf '%s' '' > "$unverf"; printf '%s' '' > "$pendf" +out=$(revout) +[ -z "$out" ] && pass "shown+pending resolves to shown (silent)" || fail "shown+pending resolves to shown (silent)" +[ ! -f "$pendf" ] && pass "the next event clears the coexistence" || fail "the next event clears the coexistence" +out=$(revout) +[ -z "$out" ] && pass "coexistence cleanup does not repeat" || fail "coexistence cleanup does not repeat" + +# 38h. shown means silent: a spent one-shot must not re-fire. +reset_all +unrec_rev >/dev/null +out=$(unrec_rev) +[ -z "$out" ] && pass "shown means silent" || fail "shown means silent (got [$out])" +[ ! -f "$pendf" ] && pass "shown means silent writes nothing" || fail "shown means silent writes nothing" + +# 38i. C2, BOTH DIRECTIONS — the accepted residual: counted, nothing delivered, nothing +# recorded. Neither direction may be quietly closed OR widened. +reset_all; : > "$offf"; mkdir -p "$pendf" +out=$(unrec_rev); rc=$? +[ "$rc" = 0 ] && [ -z "$out" ] && pass "C2/1 (suppressed): exits 0, no output" \ + || fail "C2/1 (suppressed): exits 0, no output" +[ ! -f "$unverf" ] && pass "C2/1: neither marker recorded" || fail "C2/1: neither marker recorded" +[ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "C2/1: counted silently" || fail "C2/1: counted silently" +rm -f "$offf" +# rmdir BEFORE the reset: `reset_all` calls `rm -f` on the marker paths, which prints an +# is-a-directory error to an unredirected stderr and pollutes every later run. +rmdir "$pendf" 2>/dev/null +reset_all; mkdir -p "$pendf" +run_closed "$(disc_payload unrecognized mcp__codex__review)"; rc=$? +[ "$rc" = 0 ] && pass "C2/2 (failed write): exits 0" || fail "C2/2 (failed write): exits 0 (got $rc)" +[ ! -f "$unverf" ] && pass "C2/2: neither marker recorded" || fail "C2/2: neither marker recorded" +[ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "C2/2: counted silently" || fail "C2/2: counted silently" +rmdir "$pendf" 2>/dev/null +reset_all + +# 38j. The bgAdvice one-shot. It must not burn on a message nobody saw. +bg_rev() { run "$(disc_payload backgrounded mcp__codex__review)"; } +reset_all +out=$(bg_rev) +printf '%s' "$out" | grep -q 'moved to the background at the auto-background threshold' \ + && pass "long advice is shown first" || fail "long advice is shown first" +[ -f "$bgadvf" ] && pass "long advice writes its marker" || fail "long advice writes its marker" +out=$(bg_rev) +printf '%s' "$out" | grep -q 'CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS' \ + && pass "short form names the variable" || fail "short form names the variable" +printf '%s' "$out" | grep -q 'moved to the background at the auto-background threshold' \ + && fail "short form must not repeat the long guidance" || pass "short form replaces the long guidance" +reset_all; : > "$offf" +bg_rev >/dev/null +[ ! -f "$bgadvf" ] && pass "off: does not burn the bg one-shot" || fail "off: does not burn the bg one-shot" +rm -f "$offf" +reset_all +run_closed "$(disc_payload backgrounded mcp__codex__review)" +[ ! -f "$bgadvf" ] && pass "failed write: does not burn the bg one-shot" \ + || fail "failed write: does not burn the bg one-shot" +reset_all +mkdir -p "$bgadvf" +out=$(bg_rev 2>/dev/null) +printf '%s' "$out" | grep -q 'moved to the background at the auto-background threshold' \ + && pass "bgAdvice write failure repeats the long form" || fail "bgAdvice write failure repeats the long form" +rmdir "$bgadvf" 2>/dev/null +reset_all + +# 38k. Invariant 1 at EVERY marker path, under sh and under dash. +for mk in "$bgadvf" "$unverf" "$pendf"; do + reset_all + mkdir -p "$mk" + case "$mk" in + "$bgadvf") pl=$(disc_payload backgrounded mcp__codex__review) ;; + *) pl=$(disc_payload unrecognized mcp__codex__review) ;; + esac + printf '%s' "$pl" | "$HOOK_SH_BIN" "$HOOK" >/dev/null 2>&1; rc=$? + [ "$rc" = 0 ] && pass "exits 0 with a directory at $(basename "$mk")" \ + || fail "exits 0 with a directory at $(basename "$mk") (got $rc)" + if command -v dash >/dev/null 2>&1; then + printf '%s' "$pl" | dash "$HOOK" >/dev/null 2>&1; rc=$? + [ "$rc" = 0 ] && pass "dash: exits 0 with a directory at $(basename "$mk")" \ + || fail "dash: exits 0 with a directory at $(basename "$mk") (got $rc)" + else + skip "dash unavailable — the special-builtin assertion for $(basename "$mk") did not run" + fi + rmdir "$mk" 2>/dev/null +done +reset_all +# The A5 row `flush wrote, unverified written, pending DELETE fails` has no surgical fault +# available: one permission governs both operations on .context, and a directory at the +# pending path is not seen as pending at all, so the disclosure never fires. Recorded as +# an open Gate-B obligation (P9-9) rather than asserted by a row that proves something else. +skip "A5 row 'pending delete fails' — no operation-specific fault available (P9-9)" + +# 39. FAULT TOLERANCE OF THE TWO LOAD-BEARING TOOLS. Three shapes each — absent, nonzero +# exit, partial output then failure — through both gate tools. Every shim is verified +# in BOTH directions before an assertion depends on it, and a shim that cannot be +# built skips its dependent assertions WITH it. +mk_shim() { # $1 = dir name, $2 = command, $3 = script body -> prints the dir, or nothing + _sd="$sandbox/$1"; mkdir -p "$_sd" || return 0 + printf '#!/bin/sh\n%s\n' "$3" > "$_sd/$2" || return 0 + chmod +x "$_sd/$2" || return 0 + printf '%s' "$_sd" +} +shim_run() { printf '%s' "$2" | PATH="$1:$PATH" "$HOOK_SH_BIN" "$HOOK"; } + +# 39a. awk faults. Input is a real FAILURE envelope — the one case where fail-open costs a +# real count, so `discards it anyway` would be fail-CLOSED on pass state. +awk_absent=$(mk_path noawk cat grep sed head tr git mkdir rm cp mktemp jq shasum sha1sum cksum) +awk_fail=$(mk_shim awkfail awk 'exit 3') +# The partial shim exits 2, not 1: status 1 is the locator saying "unambiguously nothing +# there", a DOCUMENTED verdict a healthy awk returns. A shim exiting 1 would be asserting +# that the hook mistrusts a legitimate answer, not that it refuses to classify from +# partial bytes, which is the property this row exists to pin. +awk_partial=$(mk_shim awkpartial awk 'printf "partial"; exit 2') +for pair in "absent:$awk_absent:absolute" "nonzero:$awk_fail:prefix" "partial:$awk_partial:prefix"; do + shape=${pair%%:*}; rest=${pair#*:}; d=${rest%:*}; mode=${rest##*:} + if [ -z "$d" ]; then + skip "awk fault '$shape' — shim could not be built; its assertions did not run" + continue + fi + # Both directions: awk must be broken under the fault PATH, and still work outside it. + if [ "$mode" = absolute ]; then + faultpath="$d" + PATH="$faultpath" command -v awk >/dev/null 2>&1 && { skip "awk fault '$shape' — awk still present"; continue; } + PATH="$faultpath" command -v git >/dev/null 2>&1 || { skip "awk fault '$shape' — git missing from the fault PATH"; continue; } + else + faultpath="$d:$PATH" + printf 'x\n' | PATH="$faultpath" awk '{print}' >/dev/null 2>&1 && { skip "awk fault '$shape' — the shim does not fail"; continue; } + fi + [ "$(printf 'x\n' | awk '{print}' 2>/dev/null)" = x ] || { skip "awk fault '$shape' — the real awk is unusable"; continue; } + reset_all + out=$(printf '%s' "$(disc_payload failure mcp__codex__review)" | PATH="$faultpath" "$HOOK_SH_BIN" "$HOOK"); rc=$? + [ "$rc" = 0 ] && pass "awk $shape: exits 0" || fail "awk $shape: exits 0 (got $rc)" + [ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "awk $shape: review counts (fail-open)" \ + || fail "awk $shape: review counts (fail-open)" + h=$(cat "$state" 2>/dev/null || echo '') + case "$h" in '' | unavailable) fail "awk $shape: stores a usable fingerprint (got [$h])" ;; + *) pass "awk $shape: stores a usable fingerprint" ;; esac + printf '%s' "$out" | grep -q 'classified at least one gate call as countable' \ + && pass "awk $shape: discloses the uncertainty" || fail "awk $shape: discloses the uncertainty" + reset_all + printf '%s' "$(disc_payload failure mcp__codex__exec)" | PATH="$faultpath" "$HOOK_SH_BIN" "$HOOK" >/dev/null + [ "$(cat "$countA" 2>/dev/null)" = 1 ] && pass "awk $shape: exec counts (fail-open)" \ + || fail "awk $shape: exec counts (fail-open)" +done +reset_all + +# 39b. sed faults. jq stays on the PATH, so field() routing does not need sed and the +# classifier is the only consumer left. Input is a real SUCCESS envelope: the blank +# test yields an empty string on a broken sed, which would read as blank and turn a +# genuine success into `no-result` — fail-CLOSED, the direction this design refuses. +sed_fail=$(mk_shim sedfail2 sed 'exit 3') +sed_partial=$(mk_shim sedpartial sed 'printf "partial"; exit 1') +for pair in "nonzero:$sed_fail" "partial:$sed_partial"; do + shape=${pair%%:*}; d=${pair#*:} + if [ -z "$d" ]; then + skip "sed fault '$shape' — shim could not be built; its assertions did not run" + continue + fi + faultpath="$d:$PATH" + printf 'x\n' | PATH="$faultpath" sed 's/x/y/' >/dev/null 2>&1 && { skip "sed fault '$shape' — the shim does not fail"; continue; } + [ "$(printf 'x\n' | sed 's/x/y/' 2>/dev/null)" = y ] || { skip "sed fault '$shape' — the real sed is unusable"; continue; } + reset_all + out=$(printf '%s' "$(payload mcp__codex__review "$(resp_success)")" | PATH="$faultpath" "$HOOK_SH_BIN" "$HOOK"); rc=$? + [ "$rc" = 0 ] && pass "sed $shape: exits 0" || fail "sed $shape: exits 0 (got $rc)" + [ "$(cat "$count" 2>/dev/null)" = 1 ] && pass "sed $shape: a success envelope still counts" \ + || fail "sed $shape: a success envelope still counts" + # P9-15: the review side carries the usable-fingerprint oracle its awk counterpart has. + h=$(cat "$state" 2>/dev/null || echo '') + case "$h" in '' | unavailable) fail "sed $shape: stores a usable fingerprint (got [$h])" ;; + *) pass "sed $shape: stores a usable fingerprint" ;; esac + printf '%s' "$out" | grep -q 'classified at least one gate call as countable' \ + && pass "sed $shape: discloses the uncertainty" || fail "sed $shape: discloses the uncertainty" + reset_all + printf '%s' "$(payload mcp__codex__exec "$(resp_success)")" | PATH="$faultpath" "$HOOK_SH_BIN" "$HOOK" >/dev/null + [ "$(cat "$countA" 2>/dev/null)" = 1 ] && pass "sed $shape: exec counts" || fail "sed $shape: exec counts" +done +reset_all + +# 40. MESSAGES AND COMPOSITION. Exact comparison on BOTH fields — a clause grep catches +# neither a negation, nor a dropped remedy, nor a reordered composition. Expected +# values are LITERAL COPIES: a golden that read the hook's own variable would agree +# with any text the hook emits. +# +# Every comparison goes through ONE helper that returns the field's ESCAPED bytes, +# because the jq-free extractor cannot decode and every message here contains quotes. +# Both emitters escape exactly backslash and quote, so the two agree byte for byte. +CTX_TERM='"},"systemMessage"' +MSG_TERM='"}' +field_of() { # $1 = document, $2 = ctx|msg -> the field's ESCAPED bytes + case "$2" in + ctx) _t=${1#*additionalContext\":\"}; printf '%s' "${_t%%"$CTX_TERM"*}" ;; + msg) _t=${1#*systemMessage\":\"}; printf '%s' "${_t%"$MSG_TERM"}" ;; + esac +} +golden() { # $1 = label, $2 = document, $3 = expected ctx, $4 = expected msg + _g=$(field_of "$2" ctx) + [ "$_g" = "$3" ] && pass "$1: ctx exact" || fail "$1: ctx exact (got [$_g])" + _g=$(field_of "$2" msg) + [ "$_g" = "$4" ] && pass "$1: msg exact" || fail "$1: msg exact (got [$_g])" +} + +G_FAILURE_CTX='Claude via Claude Code — gate hook. this Codex call returned an envelope reporting failure. Not counted as a gate pass, no review fingerprint stored, does not count toward the floor. Read error.code in the tool result. CODEX_EXECUTION_FAILED is the pinned server generic failure code and does NOT tell you whether the call started, so check the accompanying error message and any session artifacts before assuming nothing ran; a call that did start may have left work behind. CODEX_TIMEOUT means the executor gave up mid-run: re-run the SAME call with the SAME scope. Any other code, or no code at all, is unclassified: this state covers every envelope the hook managed to route AND read as carrying success false as its first property, which is not limited to the two codes named here — routing, locating the result text and the raw key spelling each gate it, and an envelope failing any of those lands in a different state instead. So re-run once with the same scope and, if it repeats, report the code and message verbatim together with the effective server name and version from claude mcp list — an unfamiliar code is itself evidence about which server answered. Never retry with a narrower instruction or a smaller range, because that would count a pass for less than the artifact or diff the gate requires. Report one line: \"gate pass discarded | error-code | started yes/no/unknown\". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the retry also fails, stop and surface that line; the operator note carries the configuration remedy.' +G_FAILURE_MSG='⚠ Codex call failed — not counted as a gate pass. If the code was CODEX_TIMEOUT, the fix is configuration and only you can apply it: raise the executor timeout for the Codex MCP server, or reduce load outside the review. Do not ask for a smaller review scope — a narrower pass is worth less than a slow one.' +G_NORESULT_CTX='Claude via Claude Code — gate hook. this gate call carried no result text the hook could read. Not counted as a gate pass, no review fingerprint stored. Treat the pass as not run and report it. Two causes produce this shape and the tool name cannot separate them: a hooks-API payload change, or a third-party tool returning empty or non-text content — which reaches the gates either through a mapping in .context/codex-gate.tools or as a server registered under the default name codex, so an absent mapping does not rule it out. Report one line: \"gate call unreadable | mapped yes/no from .context/codex-gate.tools | claude-code version\". Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. One retry per pass is the whole recovery budget under CLAUDE.md section 5. A repeat is configuration or contract, not a transient failure, so stop and surface it; the operator note carries both checks.' +# shellcheck disable=SC2016 # a literal copy of a shipped prompt; the backticks are its own +G_NORESULT_MSG='⚠ Gate call returned no readable result — not counted. Run both checks before concluding. First: does .context/codex-gate.tools map a tool name? Second: what does `claude mcp list` show as the effective server and version — not what .mcp.json says, because scope precedence can make a different entry of the same name effective. These checks narrow the cause; they do not prove it. If a mapping or a third-party server is in play, that tool may be returning empty or non-text content, which it can do legitimately: unmap it, or replace it with a server exposing exec and review. If both checks show the pinned server at its pinned version, a payload-contract change is the remaining explanation — record your Claude Code version and report it.' +G_BG_LONG_CTX='Claude via Claude Code — gate hook. this gate call was moved to the background at the auto-background threshold, 120 s by default, so its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. The original call may still be running and can still write its findings file later. If the tool result carries a task id, stop that task by it; if it carries none, wait for the call to finish. Do that before deleting that slot or re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. Do not re-run while that task is active: a late writer landing in a slot you already re-ran leaves a correctly terminated file from the wrong run, and no downstream check can detect that. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it as a setup problem rather than retrying again, reporting one line: \"gate pass discarded | backgrounded | second occurrence | CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS set yes/no\".' +G_BG_LONG_MSG='⚠ Gate pass discarded (backgrounded) — a setup gap, not a failed review. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from, then restart Claude Code: it reads the value at process start, so exporting it inside a tool shell leaves the running session unchanged. Use 0 to disable auto-backgrounding, or a positive value that exceeds your longest gate call, since a positive value shorter than the call still backgrounds it. Requires Claude Code 2.1.212 or newer.' +G_BG_SHORT_CTX='Claude via Claude Code — gate hook. this gate call was backgrounded and its result never reached the hook. The pass was discarded and not counted, and no fingerprint was stored. If the tool result carries a task id, stop that task by it; if it carries none, await the original call. Do that before re-running the pass. Before re-running, delete the target findings file for the pass and confirm it is gone — both branch files for a full Gate-B re-run, only the failed branch for a single-branch resume. The checks on that file establish its structure — terminator present, count matching, nothing but finding lines — and not which run produced it. Do not re-run while that task is active, so a late writer cannot land in a slot you already re-ran. One retry per pass is the whole recovery budget under CLAUDE.md section 5. If the call backgrounds a second time, stop and surface it rather than retrying again, reporting one line: \"gate pass discarded | backgrounded | second occurrence | CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS set yes/no\".' +G_BG_SHORT_MSG='⚠ Gate pass discarded (backgrounded) — not counted. Set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code; the full guidance was shown once earlier in this workspace.' +G_UNVERIFIED_CTX='Claude via Claude Code — gate hook. this workspace has classified at least one gate call as countable without being able to interpret its result, and attempted to record it. The counter is a mechanical tally, not a count of completed reviews: it can include calls that failed or reviewed nothing, so it can overstate them. Judge every pass on its findings artifact and discount any incomplete or unverified call, whatever the counter says. Normally said once per workspace. It repeats when its marker cannot be persisted, when two hook runs race, or when that marker is deleted by hand or by a tool that cleans .context, so treat a repeat as a marker problem rather than as new information.' +# shellcheck disable=SC2016 # a literal copy of a shipped prompt; the backticks are its own +G_UNVERIFIED_MSG='ℹ A gate call was classified as countable without inspection, and recording it was attempted. Causes with a check and a fix: a pinned-server envelope whose key order or formatting changed — compare the version in .mcp.json with the server actually serving the tools (`claude mcp list`), and pinning it back fixes it; a reworded backgrounding notice — set CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS in the environment Claude Code is launched from and restart Claude Code, using 0 to disable auto-backgrounding or a positive value exceeding your longest gate call, needing Claude Code 2.1.212 or newer; a broken awk or sed, which classification requires — check them functionally rather than by version flag, since --version is not POSIX and BSD sed exits nonzero for it on a healthy macOS: `printf \"x\\\\n\" | awk \"{print}\"` must print x and `printf \"x\\\\n\" | sed s/x/y/` must print y, each exiting 0. Causes with no user-side fix: a third-party tool whose envelope this hook cannot read — .context/codex-gate.tools names it only if it is mapped, and a third-party server registered under the default codex name reaches the gates with no mapping at all, so an absent mapping does not rule this out; where a mapping does exist, unmapping removes the gate rather than fixing the envelope; a payload the scan refused — four families land here: past the 1 Mi-unit size bound, past the 200-frame nesting-depth cap, structure the scan could not walk, and an ambiguity such as a duplicated tool_response, type or text key. Measuring the payload against those first two bounds rules them in or out; the remaining two produce this same message and the same state and are **not distinguishable from each other**, so record them as unresolved rather than guessing; and a defect in this hook parser — same situation. For either, keep the payload **locally and access-restricted**: it can contain prompts, absolute paths, review content, session identifiers and unrelated concurrent call data, so strip those before showing it to anyone, and never attach it unsanitized to a report. The list is not exhaustive: unrecognized is the terminal class, so any future unmatched shape lands here too.' + +reset_all +golden "failure" "$(run "$(disc_payload failure mcp__codex__review)")" "$G_FAILURE_CTX" "$G_FAILURE_MSG" +reset_all +golden "no-result" "$(run "$(disc_payload no-result mcp__codex__review)")" "$G_NORESULT_CTX" "$G_NORESULT_MSG" +reset_all +golden "backgrounded long" "$(bg_rev)" "$G_BG_LONG_CTX" "$G_BG_LONG_MSG" +golden "backgrounded short" "$(bg_rev)" "$G_BG_SHORT_CTX" "$G_BG_SHORT_MSG" +reset_all +golden "disclosure alone" "$(unrec_rev)" "$G_UNVERIFIED_CTX" "$G_UNVERIFIED_MSG" + +# 40b. COMPOSITION. A7: additionalContext bodies join with " — ", systemMessage bodies +# with a single space, and a carried disclosure is prefixed `Earlier: ` in BOTH +# fields so the two copies cannot disagree about which call the statement is about. +reset_all; : > "$offf" +unrec_rev >/dev/null # earn the debt without flushing it +rm -f "$offf" +composed=$(run "$(disc_payload failure mcp__codex__review)") +golden "composed failure+carried disclosure" "$composed" \ + "Earlier: $G_UNVERIFIED_CTX — $G_FAILURE_CTX" \ + "Earlier: $G_UNVERIFIED_MSG $G_FAILURE_MSG" +[ "$(ndocs "$composed")" = 1 ] && pass "composed pair: exactly one document" \ + || fail "composed pair: exactly one document" + +# 40c. A branch that emits NOTHING of its own must still flush the debt — alone, with no +# separator. A non-commit Bash PostToolUse is the reachable silent event. +reset_all; : > "$offf" +unrec_rev >/dev/null +rm -f "$offf" +silent_out=$(run '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"ls -la"}}') +golden "silent event flushes the debt alone" "$silent_out" \ + "Earlier: $G_UNVERIFIED_CTX" "Earlier: $G_UNVERIFIED_MSG" +printf '%s' "$silent_out" | grep -q ' — Earlier' \ + && fail "silent event: no separator" || pass "silent event: no separator" +reset_all + +# 40d. The longest, most punctuated string the FALLBACK escaper ever handles, both fields +# exact — and then parsed by a real JSON parser, since a well-formed-looking string +# comparison cannot tell a valid document from an invalid one. +reset_all; : > "$offf" +nojq_run "$(disc_payload unrecognized mcp__codex__review)" >/dev/null +rm -f "$offf" +nojq_composed=$(nojq_run "$(disc_payload failure mcp__codex__review)") +golden "jq-free composed pair" "$nojq_composed" \ + "Earlier: $G_UNVERIFIED_CTX — $G_FAILURE_CTX" \ + "Earlier: $G_UNVERIFIED_MSG $G_FAILURE_MSG" +if command -v jq >/dev/null 2>&1; then + printf '%s' "$nojq_composed" | jq -e . >/dev/null 2>&1 \ + && pass "jq-free composed document parses as JSON" || fail "jq-free composed document parses as JSON" +else + skip "jq unavailable — the fallback document was not parsed" +fi +reset_all + +# 40e. ONE DOCUMENT PER INVOCATION, for the five branches this change adds — the nine +# pre-existing ones are section 34. Fourteen in total; `= 1`, never `-le 1`. +reset_all; one_doc "failure" "$(run "$(disc_payload failure mcp__codex__review)")" +reset_all; one_doc "no-result" "$(run "$(disc_payload no-result mcp__codex__review)")" +reset_all; one_doc "backgrounded long" "$(bg_rev)" +one_doc "backgrounded short" "$(bg_rev)" +reset_all; one_doc "disclosure alone" "$(unrec_rev)" +reset_all + +# 40f. The unknown-tool note, BOTH fields exact. Section 20 asserts it by clause grep, +# which accepts any rewording that keeps three substrings; this pins the shipped +# bytes. DOUBLE-quoted, unlike the five constants above, because this prompt is +# built inline in the hook and legitimately contains apostrophes — so the `$` +# ending its matcher literal is escaped here rather than the quotes changed. +G_UNKNOWN_CTX="Claude via Claude Code — gate hook. Codex tool 'mcp__codex__codex' is not counted by the review gates. The gates count 'mcp__codex__exec' (Gate A, reviews TEXT) and 'mcp__codex__review' (Gate B, reviews a DIFF); your Codex server exposes a surface that cannot be attributed to one gate or the other. Passes made through it stay invisible and Gate B will keep reporting 'not run', because a gate that cannot attribute a call cannot credit it. Treat reviews run through this tool as uncounted and say so when you report gate status; do not read a satisfied count as covering them. Every remedy is a configuration change on the operator machine — installing a server, editing .mcp.json, writing a mapping file — so it is addressed to the operator in the note beside this one, and there is nothing here for you to apply. Said once per workspace, so treat a repeat as a marker problem rather than as new information." +G_UNKNOWN_MSG="ℹ Codex tool 'mcp__codex__codex' is not counted by the gates. The gates count 'mcp__codex__exec' (Gate A, reviews TEXT) and 'mcp__codex__review' (Gate B, reviews a DIFF). Fix, in order of preference. First: install the pinned mcp-codex-dev server, which exposes both — /dev-workflow:workflow-init writes it into .mcp.json. Second, only if your server genuinely has two tools that separate reviewing TEXT from reviewing a DIFF: map the names in .context/codex-gate.tools ('execTool=' / 'reviewTool='). Pointing both gates at one general-purpose tool moves the counters without either gate meaning what it says, which is a false checkmark and worse than this note. Either way the tool name must start with mcp__codex__: this hook is invoked by a hooks.json matcher of ^(Bash|Skill|mcp__codex__.*)\$, so an out-of-namespace name is either never delivered to this hook at all — the mapping looks applied and does nothing — or, for the two reserved names Bash and Skill, is delivered and hijacks a lifecycle event. Both are refused. Register the server under the name codex to place its tools there." +golden "unknown-tool note" "$(codextool mcp__codex__codex)" "$G_UNKNOWN_CTX" "$G_UNKNOWN_MSG" + +# 40g. P9-35. The hook's "an unusable sed goes to unrecognized" guarantee is POST-ROUTING +# only, and this pins the boundary. `field()` needs sed on the jq-free path too, so +# with jq absent AND sed absent the event and tool names come back empty, the payload +# routes nowhere, and classification is never reached: no class, no count, no +# disclosure. Read as unconditional, that guarantee would promise a message the hook +# has no path to emit — and invariant 1 still requires exit 0 through all of it. +nosed=$(mk_path nosed cat grep head tr git mkdir rm cp mktemp awk shasum sha1sum cksum) +if PATH="$nosed" command -v sed >/dev/null 2>&1 || PATH="$nosed" command -v jq >/dev/null 2>&1; then + skip "no-route oracle — could not build a PATH lacking both jq and sed" +else + reset_all + out=$(printf '%s' "$(payload mcp__codex__review "$(resp_success)")" | PATH="$nosed" "$HOOK_SH_BIN" "$HOOK") + st=$? + [ "$st" = 0 ] && pass "no jq and no sed: hook still exits 0" \ + || fail "no jq and no sed: hook still exits 0 (got $st)" + [ -z "$out" ] && pass "no jq and no sed: says nothing" \ + || fail "no jq and no sed: says nothing (got [$out])" + [ ! -f "$count" ] && [ ! -f "$state" ] && [ ! -f "$unverf" ] && [ ! -f "$pendf" ] \ + && pass "no jq and no sed: writes no gate-pass or diagnostic state" \ + || fail "no jq and no sed: writes no gate-pass or diagnostic state" +fi + +# 41. THE LOCATOR HAS A TIME BOUND, not only two size bounds. Gate-B pass 1 on 0.8.0 +# measured the shipped scan at 10.9 s for a 150 KB text block — ONE synchronous hook +# invocation, on a payload comfortably under the 1 Mi-unit ceiling, which is roughly +# 7x larger again. Both documented bounds are SIZE bounds; neither bounds the work, +# and a real Gate-B review result is routinely this size, so this is ordinary input +# rather than an attack. This row is the counterfactual check for the buffering +# rewrite and was written before it: against the pre-rewrite locator it fails by +# about 5x its own bound. +# +# The bound is wall-clock seconds at 1 s granularity — `date +%s` is what POSIX +# gives, and %N is not portable. 2 s is deliberately loose for a slow or loaded CI +# runner: a linear scan does this in well under a second (measured ~0.05 s), and the +# defect it has to catch is 10.9 s, so the gap swallows any granularity argument. +PERF_KB=150 +PERF_BOUND_S=2 +perf_big=$(awk 'BEGIN{ s=sprintf("%1024s",""); gsub(/ /,"x",s); r="" + for (i=0; i<'"$PERF_KB"'; i++) r = r s + printf "%s", r }') +if [ "${#perf_big}" -lt $((PERF_KB * 1024)) ]; then + fail "perf fixture: could not build a ${PERF_KB}KB body (got ${#perf_big} chars)" +else + # Built on its own line, then interpolated: a literal \" written inline inside $(...) + # inside "..." does not survive, which is this suite's documented nesting trap. + perf_body='{\"success\": true, \"summary\": \"'"$perf_big"'\"}' + perf_payload=$(payload mcp__codex__review "$(resp "$perf_body")") + perf_start=$(date +%s) + perf_cls=$(class_of "$perf_payload") + perf_el=$(( $(date +%s) - perf_start )) + [ "$perf_cls" = success ] \ + && pass "perf: ${PERF_KB}KB under-ceiling payload still classifies as success" \ + || fail "perf: ${PERF_KB}KB under-ceiling payload still classifies as success (got [$perf_cls])" + [ "$perf_el" -le "$PERF_BOUND_S" ] \ + && pass "perf: ${PERF_KB}KB payload classified within ${PERF_BOUND_S}s (took ${perf_el}s)" \ + || fail "perf: ${PERF_KB}KB payload classified within ${PERF_BOUND_S}s (took ${perf_el}s)" + + # 41b. THE ANCHOR-PREFIXED NEAR MISS — the shape the first perf fix did NOT cover, and + # the reason the row above was not enough on its own. Gate-B pass 2 found it: the + # `backgrounded` test was only GUARDED by the literal anchor prefix, so a block + # that DOES start with `MCP tool \"` and then never completes the notice still ran + # the quadratic longest-suffix expansion — 5.9 s at 150 KB, 23.1 s at 300 KB. The + # row above begins with an envelope, so it takes the guard's cheap path and is + # blind to this entirely. That is the general lesson worth keeping: a timed row + # only covers the branch its fixture reaches. + # + # Both `\n` placements, because they cost differently and only one is obvious: with + # NO `\n` the expansion scans the whole block, and with a LATE `\n` it scans to the + # newline — so a "does it contain \n" guard fixes the first and not the second. + # Measured 5.6 s and 5.5 s respectively before the bounded head. + for nm_case in no-newline late-newline; do + case "$nm_case" in + no-newline) nm_body='MCP tool \"'"$perf_big" ;; + late-newline) nm_body='MCP tool \"'"$perf_big"'\nend' ;; + esac + nm_payload=$(payload mcp__codex__review "$(resp "$nm_body")") + nm_start=$(date +%s) + nm_cls=$(class_of "$nm_payload") + nm_el=$(( $(date +%s) - nm_start )) + [ "$nm_cls" = unrecognized ] \ + && pass "perf: anchor-prefixed near miss (${nm_case}) is unrecognized, not backgrounded" \ + || fail "perf: anchor-prefixed near miss (${nm_case}) is unrecognized, not backgrounded (got [$nm_cls])" + [ "$nm_el" -le "$PERF_BOUND_S" ] \ + && pass "perf: anchor-prefixed near miss (${nm_case}) classified within ${PERF_BOUND_S}s (took ${nm_el}s)" \ + || fail "perf: anchor-prefixed near miss (${nm_case}) classified within ${PERF_BOUND_S}s (took ${nm_el}s)" + done +fi + + +# 41d. RESERVED-NAME MAPPING HIJACK (Gate-B pass 3). The mapping parser accepted any +# plausible token, and the mapped cases are tested BEFORE the native `Bash` and +# `Skill` cases — so `reviewTool=Bash` made a `git commit` COUNT a Gate-B pass +# instead of resetting the cycle, and `execTool=Skill` counted a skill invocation as +# a Gate-A pass. Both are false checkmarks in recorded state, reachable from a +# plausible typo, and this diff is what introduced the contract they contradict +# (`hooks.json` matches `^(Bash|Skill|mcp__codex__.*)$`, so only those two reserved +# names outside the namespace fire at all). Mapped names must now lie in +# `mcp__codex__*` and are otherwise ignored, like any other unusable line. +# SEEDED, not from clean. Asserting only that no counter APPEARS would pass against a +# hook that had simply stopped handling the event at all; the property is that the +# NATIVE branch still runs, so the commit must still RESET a cycle that is in flight. +reset_all +printf 'execTool=Skill\nreviewTool=Bash\n' > "$toolsf" +printf '2' > "$count"; printf '2' > "$countA"; printf 'seeded' > "$state" +run '{"hook_event_name":"PostToolUse","tool_name":"Skill","tool_input":{"skill":"superpowers:writing-plans"},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' >/dev/null +# `writing-plans` is a Gate-A RESET point, so the native branch clears the seeded count. +# That is the sharper assertion: a bumped count means the mapped case ran, an unchanged +# count would mean neither ran, and only the reset proves the native branch handled it. +[ ! -f "$countA" ] && pass "mapping hijack: execTool=Skill reaches the native Skill branch (Gate-A reset)" \ + || fail "mapping hijack: execTool=Skill reaches the native Skill branch (Gate-A reset) (countA=$(cat "$countA" 2>/dev/null))" +run '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"git commit -m x"},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' >/dev/null +[ ! -f "$count" ] && [ ! -f "$state" ] \ + && pass "mapping hijack: reviewTool=Bash still RESETS the cycle on commit" \ + || fail "mapping hijack: reviewTool=Bash still RESETS the cycle on commit (count=$([ -f "$count" ] && cat "$count"), state=$([ -f "$state" ] && echo present))" +# ...and a LEGITIMATE in-namespace mapping is untouched by the new check. +reset_all +printf 'execTool=mcp__codex__gateA\n' > "$toolsf" +run '{"hook_event_name":"PostToolUse","tool_name":"mcp__codex__gateA","tool_input":{},"tool_response":[{"type":"text","text":"{\"success\": true}"}]}' >/dev/null +[ -f "$countA" ] && pass "in-namespace mapping still counts" || fail "in-namespace mapping still counts" +reset_all + +# 41e. THE 4096-UNIT NOTICE BOUND, on both sides. The bounded head that made the +# anchor-prefixed near miss cheap also narrows the `backgrounded` class: a notice +# whose segment falls past the cutoff is counted instead of discarded, which is the +# wrong direction. Pass 3 asked for a genuine notice immediately INSIDE and OUTSIDE +# it, so the boundary is a tested fact rather than an assumed one. +# +# ONE UNIT apart, not "roughly either side". Pass 4 caught the first version padding +# 3900/4200: both rows stayed green for any cutoff in a 300-unit band, so they pinned +# a region rather than the constant. The block is `MCP tool \"` (11 units) + pad + +# `\" is still running after ` (26 units), so the segment ends at pad+37 and the two +# pads below put that at exactly 4096 and 4097. +PERF_NOTICE_IN=4059 +PERF_NOTICE_OUT=4060 +for _side in inside outside; do + case "$_side" in + inside) _padlen=$PERF_NOTICE_IN ; _want=backgrounded ;; + outside) _padlen=$PERF_NOTICE_OUT ; _want=unrecognized ;; + esac + _pad=$(awk -v n="$_padlen" 'BEGIN{s="";while(length(s) $_want" \ + || fail "notice bound/$_side cutoff -> $_want (got [$_cls])" +done +reset_all + +# 41c. P9-12, promoted from "inherited" to a CORRECTNESS bug by Gate-B pass 2, because the +# two halves combine into a malformed document. `flush_notes` ran unconditionally, so +# an event the hook could not route still reached `emit`; and the jq-free emitter +# interpolated `$event` without escaping while escaping ctx and msg. An event name +# carrying a trailing backslash therefore emitted +# `"hookEventName":"Bogus\","` — the backslash escapes the closing quote and Claude +# Code receives invalid JSON. Spec §3.3 requires no output and no state for an +# unroutable payload, which is also the fix. +# +# Asserted in BOTH runners: the routing gate is shell-independent, and the malformed +# document only appeared on the jq-free path, so testing one would have missed it. +MALFORMED_EVT='{"hook_event_name":"Bogus\"Evt","tool_name":"Bash","tool_input":{"command":"ls"}}' +for _r in normal nojq; do + reset_all; : > "$offf" + unrec_rev >/dev/null # earn the debt while suppressed + rm -f "$offf" + [ -f "$pendf" ] || fail "P9-12/$_r: setup — pending debt was not earned" + case "$_r" in + normal) mal_out=$(run "$MALFORMED_EVT") ;; + nojq) mal_out=$(nojq_run "$MALFORMED_EVT") ;; + esac + [ -z "$mal_out" ] \ + && pass "P9-12/$_r: unroutable event emits nothing" \ + || fail "P9-12/$_r: unroutable event emits nothing (got [$mal_out])" + [ -f "$pendf" ] \ + && pass "P9-12/$_r: unroutable event preserves the pending debt" \ + || fail "P9-12/$_r: unroutable event preserves the pending debt" + # Belt and braces: whatever it emits must at least be parseable. This is the assertion + # that actually failed before the fix, and it stays so a future change that re-opens the + # flush cannot re-open the malformed document silently. + if [ -n "$mal_out" ] && command -v jq >/dev/null 2>&1; then + printf '%s' "$mal_out" | jq -e . >/dev/null 2>&1 \ + && pass "P9-12/$_r: emitted document parses as JSON" \ + || fail "P9-12/$_r: emitted document parses as JSON" + fi +done +reset_all + +reset_all echo "---" [ "$fails" -eq 0 ] && { echo "all passed"; exit 0; } || { echo "$fails failed"; exit 1; } diff --git a/plugins/dev-workflow/hooks/fixtures/README.md b/plugins/dev-workflow/hooks/fixtures/README.md new file mode 100644 index 0000000..10c93dc --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/README.md @@ -0,0 +1,143 @@ +# Hook payload fixtures + +Claude Code `PostToolUse` payloads, used by `codex-gate.test.sh` to drive the gate hook +with the result shapes it has to classify. Each fixture is one payload file plus a +sibling `.response.json` holding that payload's `tool_response` array. + +These are test inputs, not documentation. A row here that disagrees with the file is a +defect in this README. + +## What each fixture exercises + +| Fixture | Tool | Class | Origin | +|---|---|---|---| +| `shape0-success.json` | `mcp__codex__exec` | `success` | capture, sanitized | +| `shape0-success-review.json` | `mcp__codex__review` | `success` | capture, sanitized — synthetic repo, see below | +| `shape1-fast-fail.json` | `mcp__codex__exec` | `failure` (`CODEX_EXECUTION_FAILED`) | capture, sanitized | +| `shape2-executor-timeout.json` | `mcp__codex__exec` | `failure` (`CODEX_TIMEOUT`) | capture, sanitized | +| `shape3-backgrounding-notice.json` | `mcp__codex__exec` | `backgrounded` | capture, sanitized | +| `collision-success-quotes-both.json` | `mcp__codex__exec` | `success` | **synthetic**, edited from `shape0-success` | +| `collision-failure-quotes-true.json` | `mcp__codex__exec` | `failure` | **synthetic**, edited from `shape1-fast-fail` | + +The two collision fixtures are synthetic **by necessity**: no real call produces a result +whose `summary` quotes both marker literals. They pin the direction that matters — a +success envelope quoting `"success": false` must stay `success`, and a failure envelope +quoting `"success": true` must stay `failure`. A first-match or last-match byte heuristic +fails on one of them. + +## What was sanitized, and how + +All five capture-derived payloads are real hook payloads — four `mcp__codex__exec` calls +made against this repository on 2026-07-30, and one `mcp__codex__review` call made against +a synthetic repository on 2026-08-02. They were edited **by hand**, never through `jq`: +passing a payload through `jq` reserializes `tool_response` and erases exactly the escape +and whitespace variants the classifier has to read, while a `jq -c '.tool_response'` +comparison would still call the result identical, because it compares semantics rather +than bytes. The review payload was sanitized by literal byte replacement with a count +assertion per substitution — the same thing, done mechanically. + +Seven field values were replaced, and nothing else: + +| Field | Replacement | Why | +|---|---|---| +| `session_id`, `prompt_id` | `00000000-0000-0000-0000-000000000000` | session identity | +| `transcript_path` | `/dev/null` | absolute path under `~` | +| `cwd`, `tool_input.workingDirectory` | `/tmp/fixture-repo` | machine layout; the second also carried a private scratchpad path | +| `tool_use_id` | `toolu_fixture` | session identity | +| `tool_input.instruction` | `probe: reply with ok` | prompt content across a trust boundary | + +`permission_mode`, `effort`, `hook_event_name`, `tool_name` and `duration_ms` carry no +machine or prompt data and are exactly as captured. `tool_input` is **kept rather than +emptied**: this repo's own gate prompts quote payload text, so a realistic `tool_input` is +what makes the decoy assertions mean anything. + +The review payload's `tool_input` carries three fields the `exec` captures do not. +`whatWasImplemented` and `additionalContext` are prompt content across the same trust +boundary as `instruction`, so they were replaced the same way (`probe: describe the +change`, `probe: report findings`). `baseSha`, `headSha` and `reviewType` are left as +captured — the two SHAs belong to a disposable synthetic repository and identify nothing. + +Replacing `tool_input` values makes two fixtures causally inconsistent with their own +result — `shape1-fast-fail`'s original failure was caused by a nonexistent +`workingDirectory`, and `shape3`'s backgrounding by a long essay prompt. Nothing depends +on that consistency; the classifier reads `tool_response` and the routing fields only. + +## What "byte-exact" covers, and what it does not + +**The claim:** for all five capture-derived fixtures, the **located `tool_response` text +block** is byte-identical to the same block in the source capture, **apart from the named +identifier redactions below**: + +| Fixture | Redacted | Original → replacement | +|---|---|---| +| `shape0-success.json` | Codex `sessionId` | `019fb3d4-…-ae5471c4a38f` → `00000000-0000-7000-8000-000000000000` | +| `shape0-success-review.json` | `specSessionId`, `qualitySessionId` | `019fc1d4-5421-…` → `00000000-0000-7000-8000-000000000001`; `019fc1d4-540f-…` → `…0002` | +| `shape3-backgrounding-notice.json` | background task id, both occurrences | `k7nsz5q4v` → `fixture01` | + +Every replacement is the same length as what it replaces and changes no structural byte, +so the escape and whitespace grammar is untouched. `shape1` and `shape2` carry an empty +`sessionId` and needed no redaction. + +The check that establishes this compares the located block from the source capture — +after applying **exactly** those substitutions — against the located block from the +fixture, and fails if any other byte moved. It also fails if a declared substitution +matches nothing, so a row cannot quietly degrade into a plain equality check. + +**What the claim does not cover:** the bytes of the payload *around* the located block. +The seven sanitized field values differ by design, and inter-element whitespace inside the +`tool_response` array outside the located text block is not pinned by this claim. + +## The response slices + +Each payload has a sibling `.response.json` holding only that payload's +`tool_response` array. They exist so the test driver can build payloads with `printf` and +`cat` alone: a machine without `jq` must fail the *hook*, not the driver, and +`sed`-extracting a multi-line array at test time would be a second parser nobody reviews. + +The cost is one duplicated representation per fixture. What pins it is a permanent +assertion in `codex-gate.test.sh` — one per fixture — requiring the payload and its slice +to **locate identical bytes**, with both locator statuses checked before the comparison. +That is a claim about the located text block, not about the whole array: two slices +differing only in inter-element whitespace would satisfy it. + +The slices were cut as a byte substring of the payload, verified by reconstructing the +payload from head + slice + tail and requiring the result to equal the original. Nothing +was re-serialized. + +## `shape0-success-review.json` — how the review capture was taken + +The other six fixtures are `mcp__codex__exec` payloads. This one covers +`mcp__codex__review` in `reviewType: full`, the mode this project's Gate B uses, whose +envelope carries `specSessionId` and `qualitySessionId` alongside `success` and whose +result text is a whole review rather than a word. + +A review result cannot be redacted after the fact — it quotes the code it read — so it was +produced against a **disposable synthetic repository** holding two invented functions and +nothing else, located at a neutral path so no scratch or session directory could appear in +the review text. The whole file was read end to end before it was staged: it references +`widget.js` and nothing else, and carries no path, no real code and no finding about real +code. + +**The capture method, and how foreign payloads were kept out.** Like the four `exec` +fixtures, this came from a stdin dump inserted into the installed hook. That hook serves +every project on the machine, and two other Claude Code sessions were live at the time, so +an unconditional dump would have written their payloads too. The dump was therefore made +**conditional** on this call's own shape — a review tool name *and* the synthetic repo path +— and verified in both directions before the real call: a foreign `Bash` payload wrote +nothing, a matching payload was captured. Every file the probe produced was audited +afterwards and all were this probe's own review calls. + +The installed hook was backed up first, with the backup's checksum status checked directly +rather than through a pipeline, and restored and byte-verified against that recorded hash +after every call. The probe workspace was deleted afterwards; the source capture is parked +outside the repository, at `.context/probe-payloads/`, alongside the four `exec` captures. + +**What follows for the claim.** Nothing special: this fixture's `tool_response` is covered +by the byte-exactness claim above, exactly like the four `exec` captures, with its two +session ids listed in the same redaction table. + +## Adding a fixture + +Add the payload, cut its slice, add both to the table above with the class it exercises, +and add its payload-vs-slice assertion to `codex-gate.test.sh`. A fixture with no slice +assertion can drift from its slice with every classification test still green. diff --git a/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.json b/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.json new file mode 100644 index 0000000..52be3c7 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"quotes \\\"success\\\": true and \\\"success\\\": false\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_EXECUTION_FAILED\",\n \"message\": \"spawn /bin/sh ENOENT\",\n \"recoverable\": true\n }\n}"}],"tool_use_id":"toolu_fixture","duration_ms":271} diff --git a/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.response.json b/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.response.json new file mode 100644 index 0000000..6f9e780 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/collision-failure-quotes-true.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"quotes \\\"success\\\": true and \\\"success\\\": false\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_EXECUTION_FAILED\",\n \"message\": \"spawn /bin/sh ENOENT\",\n \"recoverable\": true\n }\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.json b/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.json new file mode 100644 index 0000000..fbd7254 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"00000000-0000-7000-8000-000000000000\",\n \"output\": {\n \"summary\": \"quotes \\\"success\\\": false and \\\"success\\\": true\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"completed\"\n}"}],"tool_use_id":"toolu_fixture","duration_ms":9251} diff --git a/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.response.json b/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.response.json new file mode 100644 index 0000000..60dd3f4 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/collision-success-quotes-both.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"00000000-0000-7000-8000-000000000000\",\n \"output\": {\n \"summary\": \"quotes \\\"success\\\": false and \\\"success\\\": true\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"completed\"\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape0-success-review.json b/plugins/dev-workflow/hooks/fixtures/shape0-success-review.json new file mode 100644 index 0000000..ec65a8d --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape0-success-review.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__review","tool_input":{"instruction":"probe: reply with ok","whatWasImplemented":"probe: describe the change","baseSha":"0583d91fc501999056b985d985830ec659b3d153","headSha":"46ed50fa9ff3eee8c1d09e523ee8e04e3af9cc0a","reviewType":"full","workingDirectory":"/tmp/fixture-repo","additionalContext":"probe: report findings"},"tool_response":[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"\",\n \"specSessionId\": \"00000000-0000-7000-8000-000000000001\",\n \"qualitySessionId\": \"00000000-0000-7000-8000-000000000002\",\n \"review\": \"## Spec Compliance Review\\n\\nI’m using the code-review skill to inspect the exact base-to-head delta and verify it against the one-line specification. I’ll report only concrete spec deviations, with severity and confidence.\\n\\nThe reviewed range resolves cleanly and contains one commit touching only `widget.js`. The graph and the committed source both show `perimeter` as a two-argument exported function; I’m now cross-checking the delta independently for omissions or scope creep.\\n\\nI’m applying the verification-before-completion checklist before issuing the verdict, including an executable check of the exported helper rather than relying only on visual inspection.\\n\\n✅ **Spec compliant — NO FINDINGS.** `widget.js:4` adds `perimeter(w, h)` with the correct formula, and `widget.js:7` exports it alongside `area`. No missing requirements or unnecessary changes. **Severity:** None. **Confidence:** High.\\n\\n\\n---\\n\\n## Code Quality Review\\n\\nI’m using the code-review skill to inspect the exact base-to-head change, verify surrounding behavior and tests, and report only evidence-backed findings.\\n\\nThe review workflow explicitly calls for separate standards and specification passes, so I’ll run those independently and then reconcile them into your requested severity-based format.\\n\\nThe exact change is limited to `widget.js`; direct runtime assertions confirm the existing `area` export still works and `perimeter(3, 4)` returns `14`, including the zero-dimension case. I’m waiting on the two independent review passes before issuing the verdict.\\n\\n### Strengths\\n\\n`widget.js:4-7` correctly implements `perimeter(w, h)` as `2 * (w + h)`, exports it alongside `area`, preserves backward compatibility, and introduces no scope creep. Direct runtime assertions passed.\\n\\n### Issues\\n\\n#### Critical (Must Fix)\\n\\nNO FINDINGS — High confidence.\\n\\n#### Important (Should Fix)\\n\\nNO FINDINGS — High confidence.\\n\\n#### Minor (Nice to Have)\\n\\nNO FINDINGS — High confidence.\\n\\n### Recommendations\\n\\nNone required.\\n\\n### Assessment\\n\\n**Ready to merge?** Yes\\n\\n**Reasoning:** The focused change fully matches the specification and has no identified production-readiness concerns.\"\n}"}],"tool_use_id":"toolu_fixture","duration_ms":115499} \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape0-success-review.response.json b/plugins/dev-workflow/hooks/fixtures/shape0-success-review.response.json new file mode 100644 index 0000000..655423a --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape0-success-review.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"\",\n \"specSessionId\": \"00000000-0000-7000-8000-000000000001\",\n \"qualitySessionId\": \"00000000-0000-7000-8000-000000000002\",\n \"review\": \"## Spec Compliance Review\\n\\nI’m using the code-review skill to inspect the exact base-to-head delta and verify it against the one-line specification. I’ll report only concrete spec deviations, with severity and confidence.\\n\\nThe reviewed range resolves cleanly and contains one commit touching only `widget.js`. The graph and the committed source both show `perimeter` as a two-argument exported function; I’m now cross-checking the delta independently for omissions or scope creep.\\n\\nI’m applying the verification-before-completion checklist before issuing the verdict, including an executable check of the exported helper rather than relying only on visual inspection.\\n\\n✅ **Spec compliant — NO FINDINGS.** `widget.js:4` adds `perimeter(w, h)` with the correct formula, and `widget.js:7` exports it alongside `area`. No missing requirements or unnecessary changes. **Severity:** None. **Confidence:** High.\\n\\n\\n---\\n\\n## Code Quality Review\\n\\nI’m using the code-review skill to inspect the exact base-to-head change, verify surrounding behavior and tests, and report only evidence-backed findings.\\n\\nThe review workflow explicitly calls for separate standards and specification passes, so I’ll run those independently and then reconcile them into your requested severity-based format.\\n\\nThe exact change is limited to `widget.js`; direct runtime assertions confirm the existing `area` export still works and `perimeter(3, 4)` returns `14`, including the zero-dimension case. I’m waiting on the two independent review passes before issuing the verdict.\\n\\n### Strengths\\n\\n`widget.js:4-7` correctly implements `perimeter(w, h)` as `2 * (w + h)`, exports it alongside `area`, preserves backward compatibility, and introduces no scope creep. Direct runtime assertions passed.\\n\\n### Issues\\n\\n#### Critical (Must Fix)\\n\\nNO FINDINGS — High confidence.\\n\\n#### Important (Should Fix)\\n\\nNO FINDINGS — High confidence.\\n\\n#### Minor (Nice to Have)\\n\\nNO FINDINGS — High confidence.\\n\\n### Recommendations\\n\\nNone required.\\n\\n### Assessment\\n\\n**Ready to merge?** Yes\\n\\n**Reasoning:** The focused change fully matches the specification and has no identified production-readiness concerns.\"\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape0-success.json b/plugins/dev-workflow/hooks/fixtures/shape0-success.json new file mode 100644 index 0000000..f2a3c4a --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape0-success.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"00000000-0000-7000-8000-000000000000\",\n \"output\": {\n \"summary\": \"ok\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"completed\"\n}"}],"tool_use_id":"toolu_fixture","duration_ms":9251} diff --git a/plugins/dev-workflow/hooks/fixtures/shape0-success.response.json b/plugins/dev-workflow/hooks/fixtures/shape0-success.response.json new file mode 100644 index 0000000..65dbdd0 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape0-success.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": true,\n \"sessionId\": \"00000000-0000-7000-8000-000000000000\",\n \"output\": {\n \"summary\": \"ok\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"completed\"\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.json b/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.json new file mode 100644 index 0000000..68a3872 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_EXECUTION_FAILED\",\n \"message\": \"spawn /bin/sh ENOENT\",\n \"recoverable\": true\n }\n}"}],"tool_use_id":"toolu_fixture","duration_ms":271} diff --git a/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.response.json b/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.response.json new file mode 100644 index 0000000..3f95d83 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape1-fast-fail.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_EXECUTION_FAILED\",\n \"message\": \"spawn /bin/sh ENOENT\",\n \"recoverable\": true\n }\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.json b/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.json new file mode 100644 index 0000000..a1f4f5a --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_TIMEOUT\",\n \"message\": \"Execution timed out after 1ms\",\n \"recoverable\": true,\n \"suggestion\": \"Try increasing timeout or simplifying the instruction\"\n }\n}"}],"tool_use_id":"toolu_fixture","duration_ms":272} diff --git a/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.response.json b/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.response.json new file mode 100644 index 0000000..2619075 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape2-executor-timeout.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"{\n \"success\": false,\n \"sessionId\": \"\",\n \"output\": {\n \"summary\": \"\",\n \"filesModified\": [],\n \"filesCreated\": []\n },\n \"status\": \"error\",\n \"error\": {\n \"code\": \"CODEX_TIMEOUT\",\n \"message\": \"Execution timed out after 1ms\",\n \"recoverable\": true,\n \"suggestion\": \"Try increasing timeout or simplifying the instruction\"\n }\n}"}] \ No newline at end of file diff --git a/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.json b/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.json new file mode 100644 index 0000000..dc71c89 --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.json @@ -0,0 +1 @@ +{"session_id":"00000000-0000-0000-0000-000000000000","transcript_path":"/dev/null","cwd":"/tmp/fixture-repo","prompt_id":"00000000-0000-0000-0000-000000000000","permission_mode":"bypassPermissions","effort":{"level":"high"},"hook_event_name":"PostToolUse","tool_name":"mcp__codex__exec","tool_input":{"instruction":"probe: reply with ok","workingDirectory":"/tmp/fixture-repo"},"tool_response":[{"type":"text","text":"MCP tool \"codex/exec\" is still running after 120s. It was moved to the background as task fixture01 and keeps running; you'll receive a notification with the result when it completes. You can keep working in the meantime. To stop it, use TaskStop with task_id \"fixture01\". Note: it does not survive exiting this session."}],"tool_use_id":"toolu_fixture","duration_ms":120008} diff --git a/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.response.json b/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.response.json new file mode 100644 index 0000000..884854e --- /dev/null +++ b/plugins/dev-workflow/hooks/fixtures/shape3-backgrounding-notice.response.json @@ -0,0 +1 @@ +[{"type":"text","text":"MCP tool \"codex/exec\" is still running after 120s. It was moved to the background as task fixture01 and keeps running; you'll receive a notification with the result when it completes. You can keep working in the meantime. To stop it, use TaskStop with task_id \"fixture01\". Note: it does not survive exiting this session."}] \ No newline at end of file diff --git a/todos.md b/todos.md index c934c94..f0dfb8c 100644 --- a/todos.md +++ b/todos.md @@ -24,47 +24,64 @@ driven by recurrence rather than by enthusiasm. ### Parked (trigger-gated) -- [ ] **A failed Codex call counts as a pass — false ✓ in the firing direction.** - Derived while writing the 0.5.1 file-first protocol (PR #9), from a Gate-B finding - that corrected the opposite belief. The chain, each link checked against source - rather than inferred: the pinned `mcp-codex-dev@1.0.1` **catches** its own - exceptions — executor timeouts and aborts included — and *returns* - `{success: false, …}` as an ordinary result, without throwing and without setting - `isError` (`dist/tools/codex-review.js`, `codex-exec.js`). Claude Code therefore - classifies it as a **successful** tool call, so `PostToolUse` fires rather than - `PostToolUseFailure`. The hook's `PostToolUse` branch inspects nothing about the - result: for `$review_tool` it computes `tree_hash`, **stores that fingerprint**, - bumps the cycle counter unconditionally, and sets the fresh-streak counter to 0 - (fingerprint unavailable), 1 (fingerprint changed) or its prior value plus one - (fingerprint unchanged) — the streak is not a second cumulative counter; for - `$exec_tool` it bumps `countA` unconditionally. - Consequence: three timed-out Gate-A calls satisfy the Gate-A floor, and one - timed-out Gate-B call stores a current-content fingerprint for a review that read - nothing — the satisfied message then reports a fresh pass covering exactly the - content nobody reviewed. That is a false ✓ in the hook's recorded state, the - direction invariant 2 calls dangerous. - *What the shipped 0.5.1 prompts already do about it, stated so nobody over-scopes - the fix:* they classify a timeout or abort as an incomplete pass, require every - incomplete pass to be discounted **regardless of what the counter says**, and allow - one recovery attempt. So the residual defect is not "no mitigation exists" — an - earlier draft of this row claimed that and contradicted text shipped in the same - PR — it is that the mitigation is instruction-backed and depends on the agent - noticing and obeying the failed result, while the hook's own state is wrong either - way and stays wrong for anyone reading it later. - *Candidate fix, explicitly unverified:* skip the bump and the fingerprint store - when the result reports failure. The `PostToolUse` payload is documented to carry - `tool_response`, but **what it actually contains for an MCP tool on this server is - not established** — the hook has no `tool_response` reader at all today - (`input_field` parses only `.tool_input`), and the one place the hook reasons about - `tool_response` records that Bash's shape carries no exit status, which is why the - commit-reset deliberately ignores success. Verify the real payload for - `mcp__codex__*` before writing any matcher; a matcher built on an assumed shape - fails silently and in the same dangerous direction. Note also that failing closed - here is the *safe* direction for once — not counting a real pass costs a re-run, - while counting a dead one is the false ✓. - *Trigger: this session's discovery — already fired.* Deliberately not fixed in - PR #9, whose scope guard is prompts and templates only; this needs hook code and - regression tests. +- [ ] **Locator: TWO quadratic paths — `skipval`'s container walk and the record accumulator.** `substr(s,i,1)` is + O(len) per call in BWK awk, so a large VALID sibling container before `tool_response` + is quadratic: 3.2 s at 200 KB, 11.5 s at 400 KB, in one synchronous hook invocation. + Only the 1 Mi-unit ceiling stops it, and a payload just under the ceiling still costs + tens of seconds — so the ceiling is load-bearing rather than a formality. Found at + Gate B pass 2 on 0.8.0, after two other quadratics in the same scan were fixed. The + candidate fixes are a jump-based walk (linear for realistic shapes, still quadratic + for many-sibling-container payloads), a work budget scaled by payload length, or + lowering the ceiling — all three are design calls, which is why this is a row and not + a patch. **Second path, found at pass 3:** `s = s $0 "\n"` rebuilds the accumulated + input once per input line, so a newline-rich (pretty-printed) payload is quadratic in + line count independently of the container walk — 0.35 s at 4k lines, 2.69 s at 16k. + Chunked accumulation reduces but does not remove it; the two paths share a fix only if + the scan stops indexing the payload with `substr`. *Trigger: a report of a slow hook, + or any change that raises the ceiling.* +- [ ] **A5 marker matrix and A6 composition coverage are narrower than the approved plan.** + The marker-lifecycle rows run through one emitter pair rather than both, omit the + mixed pending-disclosure/background-advice write-failure combinations, and P9-9's + pending-delete-failure row is skipped by name because no operation-specific fault is + available (one permission governs both operations on `.context/`, and a directory at + the pending path is not seen as pending). A6 composition is exact-tested against a + failure message, a silent Bash event and the fallback emitter, not against every + emitting branch. Closing it needs a selective `rm` shim and per-branch composition + goldens. *Trigger: a disclosure or advice bug that the current rows do not catch.* +- [ ] **The hardening ledger has no supersession convention.** `docs/hardening-log.md`'s + header says never edit a row, and one row per hardening — so when a row's "what this + does NOT do" narration is later falsified by a feature change, there is no sanctioned + move: editing breaks the first rule and appending breaks the second. The 2026-07-20 + row now describes pre-0.8.0 counting behaviour as current. The 2026-07-20 *spec* took + a version-qualified supersession note and that worked; the ledger needs the same + convention written into its header, or an explicit "rows are historical, read the + newest row for current behaviour" statement. *Trigger: the next row falsified by a + later change — this is the second.* +- [ ] **Locator selects the `text` element by RAW BYTE comparison of `type`.** A + Unicode-escaped spelling of `text` is legal JSON meaning `text` and is not selected; + with no other element the class is `no-result` (fail-closed, so discarded rather than + miscounted, but still a wrong verdict on a legal payload). Same for escaped spellings + of the `type`/`text` keys. Characterized by a regression row and stated in spec §3.1; + closing it means decoding the `type` value for equality while still returning the + selected `text` in its original escaped bytes, since the matcher depends on those. + *Trigger: a serializer observed emitting escaped key or type spellings.* + +- [x] **A failed Codex call counts as a pass — false ✓ in the firing direction.** + **DONE in 0.8.0.** The hook now reads the result before counting. Five classes + (spec §3.3): `success` and `unrecognized` count and store a fingerprint; `failure` + (the envelope's immediately-first property is `success: false`), `backgrounded` + (the harness notice anchor at the start of the located block) and `no-result` (an + unambiguous determination that no located block yields a non-blank string) do + neither. The candidate fix recorded here was right about the direction and wrong + about the unknown: `tool_response`'s real shape for `mcp__codex__*` was established + by capturing live payloads, which now ship as fixtures. + **What remains, and it is the accepted residual, not a leftover of this row:** + locating-uncertainty is fail-OPEN — an unwalkable structure, a repeated depth-1 + `tool_response`, or a payload past the scan bounds counts, with a + once-per-workspace disclosure that the count was made without inspection. And the + counter is still not evidence: classification cannot see whether the findings file + was written, so an incomplete pass is discounted whatever the counter says. + C1–C4 in `plugins/dev-workflow/CHANGELOG.md` carry the full residual list. - [ ] **jq-free parser stops at an escaped JSON quote.** *(Candidate path, recorded 2026-08-01: the result-classification story builds a POSIX awk locator with proper string-state and backslash-parity handling. Once that exists and is proven against the @@ -192,9 +209,14 @@ backlog. profiled story before it gets a template slot. - [ ] **`/workflow-init` preflight checks `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`.** The - variable keeps a >120 s gate call in the foreground so its result reaches the hook; - without it a long call is counted at the auto-background threshold having reviewed - nothing. The result-classification story documents it in `README.md` § Setup only, + variable keeps a >120 s gate call in the foreground so its result reaches the hook. + **The failure mode this row originally described was fixed in 0.8.0** — a + backgrounded call carrying the recognized harness notice is now discarded, not + counted. What the variable still buys is the residual the CHANGELOG names as C1: + the notice is recognized *in the wording it currently uses*, so if that harness + prose ever changes the call is counted fail-open instead, with a disclosure. The + variable prevents the situation; the hook only recognizes today's spelling of it. + The result-classification story documents it in `README.md` § Setup only, deliberately — a preflight check is a second surface and was kept out of that diff. *Trigger: after that story lands* (spec: `docs/superpowers/specs/2026-07-31-failed-codex-call-counts-as-a-pass-design.md`). From 23b842d8fdc3b5869fae59c60960b681af1b26a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Mon, 3 Aug 2026 16:23:28 +0200 Subject: [PATCH 10/10] docs: correct five statements CodeRabbit found on PR #21 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five were true findings, validated by me rather than by a finding-triage subagent: this PR edits instruction-bearing paths (CLAUDE.md, AGENTS.md, plugins/, commands/), and process-pr-review's precheck forbids delegating triage there — a subagent loads the whole CLAUDE.md hierarchy, so it would review under rules the diff is rewriting. Four were introduced by the 0.8.0 change itself: - AGENTS.md listed only shellcheck and the claude CLI as prerequisites while the battery invokes `HOOK_SH=dash dash` twice. dash is now documented, with why it is named rather than pinned (system shell, not a fetched tool) and why the run matters — dropping it is how the dash-only special-builtin exit shipped. - The plan said "four script runs" for a block holding five invocations over four distinct scripts; the dash run was added and the count left stale, which is the dropped-row failure that sentence warns about. - The 2026-07-20 spec cited codex-gate.sh:361 for the PostToolUse counting branch. That was correct at the merge base — the 0.8.0 hook rewrite moved it to FAILURE_MSG. Now cited by branch name so it cannot drift again. - The story's jq parity criterion read as unqualified, but routability itself diverges for a malformed outer document, which this cycle's own sweep wrote into the design. Scoped to routable payloads, naming field() as the owner. The fifth was pre-existing but local to a row this change already edited: README claimed "a typo can't quietly unhook a gate". Verified false by running both parser guards against mcp__codex__exce — it passes the charset check and the mcp__codex__?* namespace check, so the mapping is honoured. Validation evidence — docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md battery: full quality command green — shellcheck (6 files), the hook suite 467/467 with 0 failures and 1 named skip under both sh and dash, invariants 123/123, version-bump 36/36 against the parent commit, claude plugin validate --strict. No plugins/** path is touched, so no version bump is due. check: the counterfactual for each correction is the statement it replaces — each was verified false against the artifact it describes before being rewritten, and the codex-gate.sh:361 case was checked against the merge base to establish that this change is what falsified it rather than pre-existing drift. verification (named): Gate B, 2 passes, closed clean. Pass 1's quality branch was clean; its spec branch found one Major — the README replacement said an in-namespace typo means the gate "counts nothing", turning a conditional into a categorical, since honouring the mapping only replaces the expected name and the gate would count if such a tool were invoked. Fixed to state the condition. Pass 2 returned NO FINDINGS on both branches, with the quality branch independently re-running the full battery from its own checkout. Not done in this cycle, and owed: harden-finding on these five. Both matching ledger classes — docs-drift and unverified-enforcement-claim — already sit at five occurrences, so a sixth escalates a rung rather than adding a prose row. Deferred deliberately; it changes the repository and needs its own gate. --- AGENTS.md | 9 +++++++-- README.md | 2 +- .../plans/2026-08-01-gate-pass-result-classification.md | 2 +- .../specs/2026-07-20-codex-file-first-output.md | 2 +- ...026-07-30-failed-codex-call-counts-as-a-pass-story.md | 6 +++++- 5 files changed, 15 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e0678ca..64bc11b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -251,8 +251,13 @@ Every command below was run in this session and observed to exit 0. | build | n/a — nothing is compiled or bundled | **Prerequisites and pinning.** The quality command needs `shellcheck` (0.11.0 locally; -CI runs the pinned image `koalaman/shellcheck:v0.11.0`) and the `claude` CLI (CI pins -`@anthropic-ai/claude-code@2.1.207`). Bump both deliberately, per invariant 5. +CI runs the pinned image `koalaman/shellcheck:v0.11.0`), the `claude` CLI (CI pins +`@anthropic-ai/claude-code@2.1.207`), and **`dash`** — the battery runs the hook suite +twice, once with the hook under `sh` and once under `dash`, because the hook has to be +correct under both and Ubuntu's `/bin/sh` IS dash. Bump the first two deliberately, per +invariant 5; `dash` is addressed by name because it is the system shell, not a pinned +tool. Without it the second run cannot start, and dropping that run is what let a +`dash`-only defect ship once already. **The `--exclude=SC2015` on the test file** is a single-code exclusion, not a blanket disable: every other shellcheck rule still applies to that file. Its hits are all diff --git a/README.md b/README.md index 310205d..3877c51 100644 --- a/README.md +++ b/README.md @@ -129,7 +129,7 @@ Per-workspace knobs, all files under `.context/`: | `codex-gate.floor` | a positive integer; moves the 3-passes-per-gate floor. | | `codex-gate.off` | silences the reminders; classification and state tracking keep running, so re-enabling lands on counters carrying the same semantics as gate-on — which is not the same as evidence that a review happened. | -| `codex-gate.tools` | `execTool=` and/or `reviewTool=` — counts a Codex server whose tools aren't named `exec`/`review`, and only worth it if that server really does separate text-review from diff-review; aiming both gates at one general-purpose tool moves the counters while neither gate means what it says. Each mapped name must itself lie in `mcp__codex__*`: the hook's `hooks.json` matcher is `^(Bash\|Skill\|mcp__codex__.*)$`, so an out-of-namespace name is either never delivered (the mapping looks applied and does nothing) or, for the reserved names `Bash`/`Skill`, hijacks a lifecycle event; the hook refuses both — register the server as `codex` to place its tools there. Unparseable lines are ignored, so a typo can't quietly unhook a gate. | +| `codex-gate.tools` | `execTool=` and/or `reviewTool=` — counts a Codex server whose tools aren't named `exec`/`review`, and only worth it if that server really does separate text-review from diff-review; aiming both gates at one general-purpose tool moves the counters while neither gate means what it says. Each mapped name must itself lie in `mcp__codex__*`: the hook's `hooks.json` matcher is `^(Bash\|Skill\|mcp__codex__.*)$`, so an out-of-namespace name is either never delivered (the mapping looks applied and does nothing) or, for the reserved names `Bash`/`Skill`, hijacks a lifecycle event; the hook refuses both — register the server as `codex` to place its tools there. Unparseable, out-of-namespace and reserved (`Bash`/`Skill`) lines are ignored, and the gate keeps its default `exec`/`review` name. A typo **inside** the namespace — `mcp__codex__exce` — is still honoured: the hook does not check that a mapped tool exists, so the gate now counts that name and nothing else. Whether it ever counts depends on whether a tool by that name is actually invoked; for a typo, normally never. | **Without Codex**, `/workflow-init` degrades honestly instead of scaffolding gates that can't run: it silences the hook and marks CLAUDE.md §5 `INACTIVE` with the re-enable diff --git a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md index ca96ec2..a58fdc3 100644 --- a/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md +++ b/docs/superpowers/plans/2026-08-01-gate-pass-result-classification.md @@ -85,7 +85,7 @@ Nothing here may silently evaporate. Each item names the task that discharges or ### The battery, per commit -- **Before committing** — the full `quality` row of `AGENTS.md` § Commands **minus** `check-version-bump.sh`. That is six `shellcheck` invocations, **four** script runs and `claude plugin validate . --strict` — not three, as an earlier draft said; miscounting is how `check-invariants.test.sh` or `check-version-bump.test.sh` gets dropped while the task still claims the full battery. **This block, verbatim:** +- **Before committing** — the full `quality` row of `AGENTS.md` § Commands **minus** `check-version-bump.sh`. That is six `shellcheck` invocations, **five** script runs over four distinct scripts — `codex-gate.test.sh` runs twice, once per hook shell — and `claude plugin validate . --strict`; miscounting is how `check-invariants.test.sh` or `check-version-bump.test.sh` gets dropped while the task still claims the full battery. **This block, verbatim:** ```sh shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && \ diff --git a/docs/superpowers/specs/2026-07-20-codex-file-first-output.md b/docs/superpowers/specs/2026-07-20-codex-file-first-output.md index 1984a7b..2fa8a8f 100644 --- a/docs/superpowers/specs/2026-07-20-codex-file-first-output.md +++ b/docs/superpowers/specs/2026-07-20-codex-file-first-output.md @@ -201,7 +201,7 @@ checks 1–4 failed. > that reasoning is the thing worth not losing. Counters increment in the hook's `PostToolUse` branch - (`codex-gate.sh:361`), keyed on tool name; the hook never sees the response, let alone + (the gate-tool cases in `codex-gate.sh`), keyed on tool name; the hook never sees the response, let alone the file. Claude Code fires `PostToolUse` after a tool call *succeeds* and routes a failed call to `PostToolUseFailure`, for which `hooks/hooks.json` registers no handler. **That does not mean failed reviews escape counting.** Verified in the pinned server's diff --git a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md index 2205942..3975ef2 100644 --- a/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md +++ b/docs/superpowers/stories/2026-07-30-failed-codex-call-counts-as-a-pass-story.md @@ -115,7 +115,11 @@ setting: a discarded pass reads as an actionable setup gap, not as a failed revi required reporting a third-party tool's empty response as a backgrounded call.) - [ ] A Codex call whose result reports success still counts exactly as it does today: the counter advances and, for Gate B, the fingerprint is stored. -- [ ] Behaviour on all of the above is identical whether or not `jq` is on `PATH`. +- [ ] Behaviour on all of the above is identical whether or not `jq` is on `PATH`, scoped to + payloads the hook can **route**. Routability itself diverges: a malformed outer + document does not route with `jq` present and does route through the `grep` fallback, + landing in `unrecognized`. That belongs to `field()` and predates this change (design + §3.1); classification is identical either way. - [ ] The hook exits 0 on every path above, including when the result cannot be read. - [ ] `CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS` is documented where someone setting up the workflow will see it, stating that it requires Claude Code ≥ 2.1.212 and that `0`