Record benchmark guard exposure

[MongLong0214]

Summary

• record model provenance and explicit guard exposure separately from detector outcomes
• preserve legacy M4 rows as unknown and refuse effect estimates when exposure is unknown
• make incomplete, missing, corrupt, or unexecuted guard treatment observable rather than a false no-fire

Verification

• npm run build && npm test
• targeted guard exposure, runner, provenance, and report tests
• M4 metrics output reports exposure unknown and does not compute a comparison

No benchmark data was re-run or backfilled.

[github-actions]

CommitLore — record lint

Trailers: clean — 4 commits in origin/dev..e952a48587ae7dd181f21cbe3492eacdba5e7311
Active constraints: 24 limits · 59 ruled-out · 33 warnings — from 43 records over 15 changed paths

Active constraints for the paths this PR touches

Limits (24)

• r-expreadme1 9e69abe — bench/VERDICT-M4.md still cites the Fisher figure; the two disagree until the verdict records why the number was withdrawn from the README
• r-expomerge1 d6ad014 — M4's existing rows have no exposure field and must read as unknown, not as not-exposed — backfilling by inference would erase the finding
• r-exposure52 ba69411 — legacy JSONL artifacts predate model and guard-exposure fields | their absence remains unknown and is never inferred or backfilled
• r-rdme96a 9c9371c — scripts/check-readme-numbers.mjs's withdrawal-notice and stray-statistic checks constrain what can appear outside the (absent, here) generated benchmark block — re-checked after every edit, not just at the end
• r-relinstall c6e1d04 — never tested against the real GitHub release infrastructure (no release exists yet — that is the owner's action) — verified against a locally built SEA binary, a hand-made SHA256SUMS, and a local HTTP server standing in for GitHub's release-asset redirects, which is everything this repository lets a change verify before a tag exists.
• r-seabin39 9e9cd0e — doctor's PreToolUse hook runtime check still shells to scripts/commitlore-run.sh via bash for its own probe; a binary install with the Claude Code plugin hook already wired reports a plain ENOENT-style fail there rather than trying the binary directly -- not one of B-09 · Single static binary — remove the Node runtime dependency #39's six required commands, not fixed here
• r-distrace88 d118a73 — the fix insulates bench-ablation.test.ts from the race; it does not remove the underlying design (four test files independently, redundantly rebuilding one shared dist/ in their own beforeAll). A fifth file doing the same thing, or a future check elsewhere that also depends on dist/'s mid-run stability, can still race the same way.
• r-fix70a1 d707fc7 — one encoding layer and explicit lexical forms in the four published languages; semantic paraphrases, nested encodings, and split payloads remain outside coverage
• r-shallow66 60a8659 — a depth-1 clone can only inspect its reachable commit history
• r-4c73a2 1d82a5b — code inherits the limit of any lexical surface — an alternative implemented under another name is invisible, one named in a string literal is counted; both were true of artifacts
• r-7a3e91 cf859e4 — better-sqlite3 stays external because it is native — the bundle degrades to --no-index without it, which only works because r-6f2a08 made that load lazy first
• r-9c07e2 9c4d25a — the plugin still needs Node for the CLI — the protocol does not, but guard, the index and the MCP server do (T-706 · Bundle the CLI as a single file — run from a clone alone #38)
• r-3e8a41 2f0a8a0 — scoping is not implemented, so off-path records reach every arm
• r-9c2f74 d653153 — the ablation arms cannot discriminate on these fixtures -- no-grade and no-lifecycle are byte-identical to the treatment in 9 of 10 tasks, because the seeds carry one reconstructed record and one task with a lifecycle trailer between them
• r-9c2f74 d653153 — the harness assembles its own projection rather than calling the shipped injector, so what is measured is the harness's rendering of the records, not src/core/inject.ts (issue B-08 · Replace the benchmark harness injector with the actual src/core/inject.ts #36)
• r-6e1a72 5e09846 — npx commitlore is the first thing a reader will try, and it fails until the package is published
• r-3f7a29 49817dc — reconstruction reads text written before the protocol existed, so the evidence is thinner than a harvest and the discard rate is expected to be high
• r-5c8b31 60ddc39 — the agent CLI exposes no in-flight turn limit, so a per-task turn budget can only ever be observed with this driver
• r-0b7c44 d2b2ce3 — a command is only real once --help names it, because that is where users look before they read source
• r-6c2b95 5da793c — the CLI holds no API key, so any real driver runs through the user's own agent session and cannot be exercised in this environment
• r-7f0e39 76f3f2d — literal substitution only catches the exact strings you list, so the same term written with a different separator survives
• r-9d31b7 4ac6e30 — the example lives in four translated files, so any fix that is not mechanically enforced will drift again on the next edit
• r-c0f4e2 3d249cd — npm gitlore is held by an active same-domain CLI, so the owner's first-choice name was not available
• r-a8f3c1 ef48843 — Rename must land before any code exists -- after 27 tickets it would touch spec, fixtures, index, hooks and every doc

Ruled out (59)

• r-exposure52 ba69411 — compute a treatment effect with unknown guard exposure | an old row that never recorded whether the guard reached the run cannot distinguish no treatment from an ignored treatment
• r-m4docs1 32d5bf1 — keeping the withdrawal notice and only landing the verdict document | bench/report.ts already draws this line -- a provenanced dataset that still shows a withdrawal is a hard failure in check-readme-numbers.mjs (checked here), not a style choice left open
• r-rel0200a a074754 — bumping ci.yml's "v0.1.0 was published with zero attached assets" comments | those describe a historical fact about the actual v0.1.0 release, not a version this project declares; the check they document (releases/latest/download/SHA256SUMS returning 200) is written to start exercising the real path automatically the day any release ships assets, v0.2.0 included, with no workflow edit
• r-rel0200a a074754 — touching docs/adr/ADR-0001-scope-v010.md, docs/tickets/release.md, bench/VERDICT-M1.md, HANDOFF.md, bench/ROUTE-GAP.md | planning and historical-record prose that names v0.1.0 as a past decision or measurement subject, not a live version carrier
• r-rel0200a a074754 — changing test/mcp.test.ts's CommitLore-Version: 0.1.0 fixture trailer | that's protocol-version content inside a synthetic seed commit (what an old commit's trailer looked like), unrelated to and never asserted against package.json's version
• r-rdme96a 9c9371c — dropping the git-clone / source-build paths from the top entirely | install.sh is the fast path, not a universal one (no Windows binary yet, per ADR-0015) — the detailed section has to stay reachable, just not first
• r-relinstall c6e1d04 — guessing the current version to build the asset URL directly | would need either the GitHub API (rate-limited, needs no-auth headers handled correctly) or trusting a redirect's final Location header parsing. Downloading the fixed-URL SHA256SUMS first and reading the real asset name back out of it needs neither and is what the checksum step has to fetch anyway.
• r-relinstall c6e1d04 — local for scoping — not POSIX per se, but supported by dash, bash, and every shell this script is realistically piped into (verified directly, see Verified) | not used in the end; the script has few enough variables that scoping was not needed, only noted here because it was considered.
• r-seabin39 9e9cd0e — mainFormat: "module" (an ESM SEA main) | verified to fail both blob generation and runtime on this Node line, not merely documented as unsupported (see above)
• r-seabin39 9e9cd0e — pkg / nexe | third-party bundlers embedding a separate, forked Node runtime this project does not control the patch cadence of; pkg is archived upstream. Trades the Node runtime dependency this ticket removes for a different, less-maintained one
• r-seabin39 9e9cd0e — Deno compile / Bun compile | a different runtime. node:sqlite, the TypeScript, and NodeNext resolution are all Node-specific; retargeting them is a second runtime port, not a build step, and issue B-09 · Single static binary — remove the Node runtime dependency #39's own first option ("Node SEA -- no source rewrite") needs none
• r-seabin39 9e9cd0e — reimplement in Go/Rust | issue B-09 · Single static binary — remove the Node runtime dependency #39's own second option, and a real one via spec/fixtures + spec/contract-cases, but an order of magnitude more work than this ticket and not needed to solve either problem (latency, no-Node-on-PATH) this ticket opens with
• r-seabin39 9e9cd0e — committing dist/commitlore next to dist/commitlore.mjs | breaks ADR-0011's committed-dist/-matches-src/ invariant at ~115 MiB per platform/arch, and a pushed blob that size is not removable from git history again
• r-seabin39 9e9cd0e — Windows (commitlore.exe) in this PR | Node's docs describe a signtool path this repository has no CI runner to verify; shipping an unverified platform claim is what this project's numbers-or-silence discipline exists to refuse. classifyBinTarget and the resolution order are written so it is a small additive follow-up, not a redesign
• r-distrace88 d118a73 — consolidating the four files' redundant tsc rebuilds into one vitest globalSetup | would remove the race for every file at once and is the more complete root-cause fix, but touches five files' bootstrap plus global config, and mcp.test.ts's beforeAll already tolerates unrelated in-flight tsc errors elsewhere in src/ (OWNED_SOURCES filtering) in a way a single shared build step cannot preserve without its own redesign — bigger blast radius than this issue asked for.
• r-distrace88 d118a73 — raising bench-ablation's timeout, globally or per-test | proven not to be a timeout problem at all: execFileSync blocks synchronously, so vitest's timeout mechanism cannot even engage against it (verified with a probe test); the actual failure is the runner subprocess legitimately exiting 1.
• r-fix70a1 d707fc7 — exhaustive per-language phrase enumeration | unbounded phrase lists cannot provide semantic coverage, so this fix documents a bounded lexical policy and independent corpus
• r-6f92c4 3cebb89 — restating the withdrawn numbers as prose ("we measured a reduction") | it is the same claim with the evidence removed
• r-6f92c4 3cebb89 — leading with the protocol's features and putting the measurement record near the bottom | that is the arrangement of someone hoping it is not read
• r-2b58d4 4842356 — exempting datasets written before the fields existed | it is one line and it deletes the guarantee
• r-8e15c2 70f6bcf — hashing only the files the entry point transitively imports | it is the same judgement call that produced this bug, made with more machinery
• r-6b83f2 4c1a503 — deleting both sentences | the clone-runs-without-installing claim is true for validate, context, guard and the MCP server, and dropping it would understate what a clone gives you as badly as the old text overstated it
• r-3d92a8 f85101a — keeping the searches first and fixing the shim | the shim belongs to npm, not to us, and the version-skew problem survives the fix
• r-3d92a8 f85101a — a config-only hook check | it was written, it reported ok, and the hook failed on the next commit
• r-4a17d9 68d4c92 — guard versus the control as the primary test | it would confound the route with whether records exist at all, and the interesting question is which route works better once they do
• r-4a17d9 68d4c92 — a blocking guard arm | GUARD-CANNOT-BLOCK measured true and false positives in one score band; the only precision-safe threshold catches one in five
• r-3d61a8 b9f34b3 — adding --ablate-* flags to the shipped inject command | it would create exactly the path from user input to an ablation that the design forbids, and the one that matters injects a prompt-injection payload
• r-3d61a8 b9f34b3 — importing src/core/inject.ts from bench | src/ uses NodeNext .js specifiers, so --experimental-strip-types cannot load it. dist/ is committed (ADR-0011) and is always present
• r-3d61a8 b9f34b3 — letting the shim grow into a second injector | that is what bench/context.ts became, and then the benchmark measured the copy
• r-4c73a2 1d82a5b — re-scoring the recorded transcripts and reporting the result | the labels would be chosen with the outcome known, which §4 forbids; the recorded runs were used only to check the surface behaves as intended
• r-4c73a2 1d82a5b — moving violation_if to code as well | it is instrumented-only, and changing an unregistered metric alongside a registered one makes it unclear which change any difference belongs to
• r-7a3e91 cf859e4 — inlining spec/SPEC.md and the schema into the bundle | SPEC.md would need a codegen step that itself needs a drift guard, and the package-root walk removes the reason to want it
• r-7a3e91 cf859e4 — replacing the tsc output with the bundle | test/cli.test.ts, test/hooks.test.ts and test/mcp.test.ts import dist internals by path
• r-c53d19 110be8c — leaving the claim and letting T-706 · Bundle the CLI as a single file — run from a clone alone #38 make it true later | the README is what someone reads while deciding to adopt this, and a claim that is false today does not become honest because it is scheduled
• r-9c07e2 9c4d25a — invoking npx on every Edit | it puts a registry round trip on the hot path of every tool call, which is how a hook earns being uninstalled
• r-9c07e2 9c4d25a — committing dist/ so the plugin is self-contained from a git clone | it puts build output in review diffs forever to save one background install
• r-0d4b81 8005227 — a longer quickstart that demonstrates context, limits, ruled-out, warnings and stale | an agent calls those itself once the MCP server is registered, so listing them teaches the human a workflow that is not theirs
• r-7f31c9 750ab17 — reporting both datasets from one source list | readSources groups by condition and cannot separate repositories, so any second dataset with a commitlore-on arm silently corrupts the headline test
• r-6d92c4 b058de2 — documenting grep -a as the convention | it leaves every future reader one forgotten flag away from the same silent empty result, and the cause takes six characters to remove
• r-3e8a41 2f0a8a0 — off-path records that advocate the ruled-out option | with no scoping they reach all three arms, raising re-proposal everywhere and compressing the grading and lifecycle contrasts the set is built to isolate
• r-9c2f74 d653153 — resume the pilot into the same file | a new process would load the edited code and create the mixing that had not happened
• r-9c2f74 d653153 — run the ablation arms as they stand | three nulls from comparing identical inputs read as "these guarantees do not matter"
• r-9c2f74 d653153 — keep the tasks that showed an effect and rewrite only the rest | the property is the criterion, not the direction of the result
• r-6e1a72 5e09846 — leave the banner until release | it understates for weeks and readers leave rather than build from source
• r-6e1a72 5e09846 — update English only and translate later | the lag is itself a wrong answer for whoever reads the other three
• r-3f7a29 49817dc — repair a draft that fails verification | backfill's source material is weak enough that a repair loop would mostly be inventing
• r-3f7a29 49817dc — write reconstructed records into commit messages | history rewriting is irreversible and reaches every existing clone
• r-3f7a29 49817dc — post a fresh comment per push | it turns the signal into noise and the check gets muted
• r-5c8b31 60ddc39 — keep turns and explain it in prose | the JSONL outlives the prose, and whoever reads the rows later will not have it
• r-5c8b31 60ddc39 — drop the turn budget since it cannot be enforced | the overrun is still the signal that a run went off the rails
• r-6c2b95 5da793c — skip the dry-run driver | then nothing exercises the harness until a key exists, and the first real run debugs the harness instead of measuring anything
• r-6c2b95 5da793c — emit dry-run rows without a marker | indistinguishable from measurements the moment they leave the terminal
• r-9d31b7 4ac6e30 — fix the values and move on | the same drift already happened once through a rename, and prose review did not catch it either time
• r-9d31b7 4ac6e30 — parse the README at runtime in the CLI | the check belongs in the conformance suite, not in shipped code
• r-c0f4e2 3d249cd — GitLore published as git-lore | the binary and search results still collide with the existing gitlore tool
• r-c0f4e2 3d249cd — keep Annals | the sound problem does not decay, and with code near zero this is the cheapest moment the project will ever have
• r-c0f4e2 3d249cd — rename code and spec first, documents later | the drift window makes every artifact written in it wrong
• r-a8f3c1 ef48843 — keep name, change vocabulary only | vocabulary is the protocol, so half the change leaves the substance untouched
• r-a8f3c1 ef48843 — drop Certainty as a dead field | a real route exists -- stale sweep prioritizes guess-level records for review

Warnings (33)

• r-exposure52 ba69411 (claim) — guard exposure is instrumented in the benchmark hook adapter, not by changing guard scoring | M4 rows remain unexposed and metrics refuses their effect estimate
• r-rel0200a a074754 (claim) — scripts/commitlore-bootstrap.sh is orphaned -- no hooks.json entry invokes it, and its npm-install strategy contradicts ADR-0011. It still carries a live version default, now bumped for consistency, but nothing exercises it. Worth a follow-up issue: either wire it up correctly or delete it.
• r-seabin39 9e9cd0e (claim) — node:sea is "Active development" per Node's own docs; its schema or CommonJS-only constraint could change between Node versions. core/paths.ts's readInstalledFile/isSea split and build-binary.mjs's asset map are the one place that assumption is absorbed, same posture as ADR-0012 already committed to for node:sqlite
• r-distrace88 d118a73 (claim) — COMMITLORE_BENCH_DIST_DIR is read once at module load (bench/hooks-settings.ts DIST_DIR is a top-level const) — set it before importing the module, not after.
• r-fix70a1 d707fc7 (claim) — add malicious and benign fixtures together when extending scanner patterns; false positives can make the defence unusable
• r-shallow66 60a8659 (claim) — shallow history remains advisory; query and guard exit-code semantics are unchanged
• r-6f92c4 3cebb89 (claim) — with no numbers, the first-impression case now rests entirely on the test links. If M3-b also comes back null, this framing is what the project has
• r-2b58d4 4842356 (claim) — this leaves the README with no measured numbers at all until M3-b runs. That is the honest state and it is also a worse first impression. The alternative was publishing numbers produced by a binary nobody recorded
• r-8e15c2 70f6bcf (claim) — this is the second time in two commits that a fix for the M3 failure did not actually cover the M3 failure. The first was caught by testing the fix against the incident rather than against its own tests
• r-6b83f2 4c1a503 (claim) — the second claim will become true when ADR-0012 lands and false again if the notes refspec story changes. A README sentence about distribution needs a test, and there is none — scripts/check-readme-numbers.mjs checks numbers
• r-3d92a8 f85101a (claim) — hook-runtime executes the hook on every doctor run. The probe message is valid so nothing is written, but it is no longer a read-only command
• r-3d92a8 f85101a (claim) — the check pins PATH to /usr/bin:/bin, which assumes git is there. On a system where it is not, this reports a hook failure that is really a probe failure
• r-4a17d9 68d4c92 (claim) — both hooked arms pay a subprocess per edit. M2 put all three of its timeouts in its hooked arm, and a timeout truncates a run, which flatters whichever arm suffers it — stated before the numbers exist
• r-3d61a8 b9f34b3 (claim) — this changes what the harness measures, so M1-b's numbers do not carry forward. A run under this delivery path is a new measurement and needs its own registration
• r-4c73a2 1d82a5b (claim) — SURFACES in task-loader.ts and MatchSurface in types.ts are two lists of the same thing — adding a surface means editing both
• r-7a3e91 cf859e4 (claim) — hardcoding ../ counts back to the package root is what broke this — new code reads assets through installedPath(), never through import.meta.url
• r-c53d19 110be8c (claim) — r-6f2a08's message says the clone gap closes with B-09 · Single static binary — remove the Node runtime dependency #39; it closes with T-706 · Bundle the CLI as a single file — run from a clone alone #38. The commit message is history and stays as written
• r-9c07e2 9c4d25a (claim) — hooks here must exit 0 on every path — a non-zero exit from a PreToolUse hook blocks the edit, and no record is worth that
• r-0d4b81 8005227 (claim) — claude mcp add commitlore -- commitlore mcp is Claude Code's syntax — other MCP clients register a stdio server their own way
• r-7f31c9 750ab17 (claim) — adding a file to README_SOURCES pools it into every aggregate in the block, including the significance test — check the arm names first
• r-6d92c4 b058de2 (claim) — keep the NUL separator when touching these keys — a printable delimiter can appear inside a trailer value and would collide
• r-8b41e6 0bbad66 (claim) — the payload share and the fixture share are not interchangeable — the payload adds framing and drops what grading and lifecycle withhold, so a reader who swaps one for the other will be wrong by a small, plausible margin
• r-3e8a41 2f0a8a0 (claim) — the ablation set is a different synthetic repository from bench/tasks, so its commitlore-on arm must never be pooled with the primary matrix's
• r-9c2f74 d653153 (claim) — after the measurement, check that git status is clean and the recorded sha is still HEAD -- an edit mid-run breaks reproducibility silently, and that check is the only thing that catches it
• r-6e1a72 5e09846 (claim) — the moment v0.1.0 is published this banner is wrong again -- it names npx not working, which release makes false
• r-3f7a29 49817dc (claim) — every backfilled record is Provenance: reconstructed, which the trust model always renders as a claim -- do not add a path that lets a draft override that field
• r-5c8b31 60ddc39 (claim) — a driver that gains a real turn limit should stop emitting over-turns and start emitting an enforced label -- do not reuse over-turns for something the harness actually stopped
• r-6c2b95 5da793c (claim) — never cite a row with simulated:true -- README numbers come from bench/results logs and those rows are not results
• r-7f0e39 76f3f2d (claim) — when retiring a command name, grep the bare word too, not just the prefixed form -- the prefix is what the substitution keyed on
• r-9d31b7 4ac6e30 (claim) — the four READMEs must keep the example block byte-identical -- translating the code block will fail spec/verify.sh
• r-c0f4e2 3d249cd (claim) — ADR-0008 and ADR-0009 keep the literal string Annals on purpose -- mechanical substitution there destroys the decision trail
• r-c0f4e2 3d249cd (claim) — the residual grep for lore_query reports a false positive because commitlore_query contains it as a substring, so check the prefix
• r-a8f3c1 ef48843 (claim) — docs/adr/ADR-0008 is the canonical vocabulary -- do not reintroduce old terms from memory

git log --follow accepts exactly one pathspec, so renames are not followed for 15 paths; query one path at a time to follow its rename chain

withheld the content of 2 record(s) graded blocked: a Ruled-out, Verified trailers matching an injection pattern is reported, never quoted (SPEC §7)

_{Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.}