The QA phase was reporting success while measuring nothing
Every run printed "QA Phase completed". Behind that line, the numbers it exists to produce were never reaching the gate that decides whether your work is good enough to advance.
None of this could show up as a failing test — a component that silently does nothing produces a green run, not a red one. It was found by reading the path from qa-lead through the Stop handlers to gate-manager end to end, and every claim below was reproduced before it was fixed.
What was actually broken
No Stop handler had ever seen its hook payload
unified-stop reads the payload with a reader that destroys stdin on resolve — deliberate, from the fix for the 15-minute hook stall in issue #139 — and then dispatches the per-agent handler with require(), in the same process. Each handler reads stdin again for itself, by which point the bytes are gone.
$ echo '{"hook_event_name":"Stop","transcript_path":"/tmp/t.jsonl"}' | node parent.js
PARENT saw: {"hook_event_name":"Stop","transcript_path":"/tmp/t.jsonl"}
CHILD saw: {}
Six metric-collecting handlers had been reading {} since the day that reader was introduced. The payload is now handed on to the later reader, and handlers resolve it to the assistant's reported text via transcript_path.
Handlers were pattern-matching the wrong text entirely
| handler | what it was reading |
|---|---|
qa-phase-stop |
called .match() on a parsed object — TypeError on its first pattern, swallowed by its own catch. M11–M15 were never written on any run since v2.1.1. |
analysis-stop, qa-stop |
their own hardcoded guidance string ("Next steps: 1. Save report to…"). M2 was always the 75 baseline; M5 was always 0 — "no errors found", every session, measured or not. |
gap-detector-stop, iterator-stop, pdca-skill-stop |
the hook envelope — hook_event_name, session_id, transcript_path, cwd — which contains none of the signals they look for. |
All six now read the agent's report.
The QA gate could never pass
gate-manager's qa gate has required qaCriticalCount === 0 since v2.1.1. No metric ID ever produced that name, and an absent metric does not satisfy a condition — so the pass count fell short on every run, and QA was structurally unable to advance to Report regardless of how your tests went.
Fixed by supplying the missing metric (M16, QA Critical Count). The gate itself is untouched: the threshold was never the problem.
Browser tests were skipped 100% of the time
Chrome MCP detection read process.env.MCP_SERVERS — a variable Claude Code does not set. It was false on every run, so L3, L4 and L5 never executed, while qa-lead's own documentation described that skip as normal fallback. A permanently dark half of the test matrix read as a feature.
Detection is now layered, most authoritative first: a runtime probe qa-lead records after actually calling a Chrome MCP tool, a BKIT_CHROME_MCP=1|0 operator override, the old environment variable, and finally your MCP config files.
A QA failure never went back to QA
act → qa (QA_RETRY) was defined, and emitted by nothing. A rejected feature dropped into the ordinary act → check loop and never returned to the phase that rejected it. Its retry counter also read and rewrote the same value, so the loop-breaker could not fire however many times a feature went round.
An unmeasured rate was recorded as 0%
isMeasured(0) is true, so a parse failure that fell back to 0 passed every "did we measure this?" check downstream — including the efficiency calculation, where improvement = 0 - previous wrote down a regression that never happened.
What changes for you
| Before | Now |
|---|---|
| "QA Phase completed" with no numbers behind it | M1–M16 carry real readings from what the agents reported |
qa gate could not return pass |
The gate passes or fails on your actual results |
| L3–L5 browser tests silently skipped, always | They run when Chrome MCP is connected, and say so when it isn't |
A QA failure quietly rejoined the act → check loop |
It returns to QA, bounded by guardrails.loopBreaker.maxQaRetries (default 3) |
A rate nobody measured displayed as 0% |
It reads "not measured" — and never counts as reaching a threshold |
qa-test-planner could not write a test plan |
It writes docs/05-qa/{feature}.test-plan.md, and the generator stops if it's absent |
qa-lead claimed four coordinated agents, dispatched three |
qa-monitor runs, so test outcomes have runtime log evidence behind them |
| The pre-release scanner scanned bkit, not your project | It scans $CLAUDE_PROJECT_DIR; --root DIR to point elsewhere, --self for the old behaviour |
Nothing here requires action from you. Update the plugin and the QA phase starts reporting what it measured.
One thing to know if you're on Claude Code v2.1.233
v2.1.233 withdrew the Todo/Task tool family — TaskCreate, TaskGet, TaskUpdate, TaskList, TodoWrite — from Opus 4.8 / Sonnet 5 / Fable 5 / Mythos 5+, keeping it for Haiku. CLAUDE_CODE_ENABLE_TODO_TOOLS=1 restores it.
This surfaced here because bkit's live hook harness reported TaskCreated / TaskCompleted as dead hooks. They are not dead — there was simply no tool for the model to call. Isolated against the hook dispatch ledger rather than against what a model says it has: all four of the harness's isolation flags were bisected one at a time and none of them moved the result, while the same trigger under the full flag set fires both hooks with that variable set. The harness now sets it, and live hook coverage went 21/23 → 23/23 with the isolation intact.
If you rely on bkit's TaskCreated hook or on skills that instruct the model to call TaskCreate, they are inert on v2.1.233 unless you set that variable. Recommended Claude Code runtime remains v2.1.220.
Verification
- 5,355 test cases · 0 failures (5,350 pass, 5 skip) — 78 new cases across three regression suites
- Full live QA 145/145 against a real Claude Code v2.1.233: 44 skills, 34 agents, 21 hook events, 19 MCP tools, and fork mode — zero failures
- Live hooks 23/23, with the L6 evidence artifact re-recorded against the shipped
hooks.json - The F-0 payload defect was reproduced on
mainand confirmed fixed on the branch with an independent two-file harness
Also in this release
The philosophy docs described ten metrics where the code has had M11–M15 since v2.1.1 — and a different ten, whose M1 was "Plan accuracy" where the code's M1 is Match Rate. Anyone matching a metric ID against a runtime value was reading two unrelated lists. That table now comes from METRIC_SPECS.
Full detail: CHANGELOG.md · PRs: #151, #152, #153, #154