Hook delivery path has no exposure instrumentation, so M4 is unanswerable

[MongLong0214]

No run in M4 received injected context. Neither arm. All 112.

bench/results/transcripts-final:  30/60  have injected_context
bench/results/transcripts-m4:      0/112 have injected_context

Broken out by arm, every one of the 112:

commitlore-guard:  injected_context present 0 / absent 56
commitlore-on:     injected_context present 0 / absent 56

The field is not vestigial — the earlier transcripts-final set populates it on exactly half its runs, which is what a two-arm design should look like. M4 populated it on none.

What this means for the verdict

bench/VERDICT-M4.md reports a null on commitlore-guard vs commitlore-on, both arms of which are defined by receiving records. If neither arm received any, the two arms were identical and the experiment compared nothing against nothing.

A null from that is not a weak result about the guard. It is not a result about the guard at all.

I am not yet asserting the delivery failed — this could be a recording gap rather than a delivery gap, and those are different failures with the same evidence. That ambiguity is itself issue #113's subject, and it is the reason #113 made metrics.ts refuse an effect estimate over rows whose exposure is unknown. That refusal now looks like it was protecting against exactly this.

Two things point at genuine non-delivery rather than a recording gap:

• matched is an empty list on every M4 transcript
• accepted_records is 1 on all 112 rows, and identical across arms — if the arms differed in what they received, that field is the one that should have differed

What has to be established, in order

1. Whether the context was delivered and not recorded, or never delivered. The harness commit is pinned (081d858c) and dist_digest is a single value across all rows, so the exact code that produced these runs is recoverable. Re-run one task, one seed, one arm at that pin and inspect. One run answers it.
2. If it was never delivered, bench/VERDICT-M4.md must say so — not as a caveat but as the headline. The verdict currently attributes the null to the guard's effect; it would instead be a harness fault, and the corrected paired/clustered analysis in that document would be analysing noise.
3. A guard against the class, since this is now the fourth instrument failure in a row (M1/M2 empty, M4 saturated, M5's registered outcome zero-variance, and now this). Before a benchmark's results are analysed, assert that the treatment arm's exposure differs from the comparator's. If it does not, the run is void, and it should be void at analysis time rather than after publication.

Provenance

Found while enumerating candidate outcome variables for M5 under issue #121, by opening the transcripts rather than the summary rows. The summary rows do not carry injected_context at all, which is why every prior analysis — including mine — missed it.

[MongLong0214]

Confirmed: non-delivery, not a recording gap

I restored the exact harness M4 ran on — git worktree add /tmp/m4v 081d858c, npm ci, npm run build && npm run bundle — and ran one probe under M4's own conditions:

node bench/runner.ts --task qualification-gitseed-approved-bool \
  --cond commitlore-guard --seed 1 --driver claude-headless --model haiku \
  --save-transcripts tprobe

Result:

cond:              commitlore-guard
injected_context:  None
matched:           []
reproposed:        False

That reproduces M4's state, so it is not a fluke of the original run. Then the decisive check — searching the agent's own transcript for any trace of the records that should have been delivered:

transcript length: 997
"commitlore"      → False
"Ruled-out"       → False
"Limit:"          → False
"active records"  → False
"r-"              → False

Nothing reached the agent. A recording gap would leave the context in the transcript and only the field empty. The transcript is clean, so the treatment was never applied.

Consequence

M4's two arms were identical. commitlore-guard and commitlore-on are both defined by receiving records; neither received any. The comparison was nothing against nothing, and its null says nothing about the guard.

bench/VERDICT-M4.md currently attributes the null to the guard's effect and analyses it with McNemar, ICC and a design effect. Those statistics are correct arithmetic on data that measured no treatment. The verdict needs a headline correction, not a caveat.

The corrected analysis I added to that verdict — the paired test, the effective n of 13, the four saturated tasks — remains true about the data, and irrelevant as evidence about the guard. Both things need saying.

What this does not change

M4's provenance is still clean: single harness commit, single dist digest, 112 rows, no mid-run rebuild. The failure is upstream of provenance — the harness recorded faithfully what it did, and what it did was run both arms without records.

Order of work

1. Correct bench/VERDICT-M4.md. The null stands as an observation and is withdrawn as evidence about the guard.
2. Fix the delivery path in the harness, with a test that fails when a treatment arm receives no context.
3. Add the analysis-time assertion from this issue's third point: a run whose treatment-arm exposure does not differ from its comparator is void, and void at analysis rather than after publication.
4. Only then is M5 worth registering. Issue The registered M5 primary outcome does not count re-proposals and is always zero #121's zero-variance outcome and this are the same lesson from two directions — the instrument must be shown to work before the experiment is run, not inferred from the fact that it produced numbers.

[MongLong0214]

Correction: I was wrong about non-delivery

My earlier comment on this issue asserted that nothing reached the agent and that delivery was confirmed broken. That was not established, and the reasoning behind it does not hold. Three things overturn it.

injected_context: null is by design, not a symptom. bench/runner.ts sets it that way for any arm that delivers through the shipped hook injector:

// shipped injector per edit, path-scoped, the way the product does;
// without one the arm falls back to the harness's session-start block.
const injectedContext =
  settings.settingsPath === null ? assembleContext(workspace.dir, condition) : null;

A null field is the signature of the hook path. bench/results/transcripts-final populates the field on 30 of 60 because it used the prompt-prepend path. Both are correct behaviour, and the contrast I read as a regression is a difference in delivery mechanism.

My decisive check could not decide anything. I searched the saved transcript for record text and found none. But the transcript field holds only the final assistant message — around 1.5 KB — not a conversation log. Injected context could never appear there. The search proves nothing in either direction.

M4 has no exposure field at all. Every one of the 112 rows in t702-m4-final.jsonl carries accepted_records and nothing else about exposure. guard_exposure and readGuardExposure were added afterwards, by #113. M4 ran before the instrument that would have recorded whether the hook fired existed.

What the defect actually is

Not a delivery bug. The hook delivery path had no exposure instrumentation, so a run that delivered nothing was indistinguishable from one that delivered. M4 is unanswerable for that reason, and its rows cannot be retrofitted because the field is absent rather than empty.

What changes

The conclusion in bench/VERDICT-M4.md stands — M4's null is not evidence about the guard — but its stated basis is wrong and is being replaced on branch bug-issue-122-basis. The correct basis is unverifiability: we did not show the treatment was absent, we showed the experiment cannot demonstrate it was present. Those are different claims and only the second one is supported.

Work on this branch is rescoped accordingly:

• keep the assertDifferentialExposure refusal in bench/metrics.ts, extended so that a dataset with the field absent on both arms is refused as unknown rather than passed through
• add the end-to-end test that a hook-plan arm records exposure showing the shim actually executed, asserted on execution rather than on the field being present
• show that test failing against a deliberately broken shim before it passes against the real one

Why I am recording my own error here

The mistake is the same one this repository has now found nine times in other people's code: a check answered without stating what it could not see. I searched a field that structurally could not contain the thing I was looking for, and reported the absence as a finding. The instrument that would have answered the question honestly is the one #113 added, and it did not exist yet when M4 ran.