What I found
commitlore-on runs the real injector (commitlore inject --hook-input), and the ablation arms run buildInjection directly through the shim in bench/hooks-settings.ts. Neither passes --trusted-author, and src/commands/inject.ts defaults trustedAuthors to [].
gradeRecord is fail-closed on an empty list — "no trusted authors are configured, so X is an outside contributor" — so every record in every benchmark arm grades claim.
Measured, not inferred. Same seeded repo, one authored record, buildInjection both ways:
--- commitlore-on (real grading, no trustedAuthors) ---
Ruled-out
[claim] r-probe01 a4613ed5 shared Redis cache | ops refuses another stateful dependency
--- no-grade ---
Ruled-out
[directive] r-probe01 a4613ed5 shared Redis cache | ops refuses another stateful dependency
Two consequences
1. The directive path is unmeasured. The plugin instructions tell agents that [directive] means "treat as a constraint", and that is the routing the product's value claim rests on. No arm has ever produced one. M1 and M5 measured an agent reading records tagged [claim] throughout.
2. bench/types.ts describes no-grade incorrectly. It says:
Inert on a task whose records are all Provenance: authored, because there is nothing to promote.
That is true of bench/context.ts's assembleContext, which routes on record.provenance === "authored". But assembleContext is only reached when writeArmSettings returns settingsPath: null, which happens only when an arm has neither a preToolUse nor an ablation — so not for no-grade. On the real path no-grade flips every record from claim to directive, on the existing fixtures, today.
The comment matters because it is load-bearing: I wrote it into bench/PREREGISTRATION-M6.md §4 as the reason M6 needs new fixtures before it can run at all. It does not.
What this needs
- Decide whether the benchmark should configure a trusted author. It is a design choice, not an oversight to patch quietly: an arm with one measures the product as a repository actually configures it, and an arm without one measures the fail-closed default. Both are defensible and they answer different questions.
- Whichever is chosen, record it in the arm definition so a reader of a result knows which was measured.
bench/README.md and any published number derived from M1/M5 should say that those runs measured claim-tagged records.
Related: #412 (M6), and bench/PREREGISTRATION-M6.md §4 needs correcting either way.
What I found
commitlore-onruns the real injector (commitlore inject --hook-input), and the ablation arms runbuildInjectiondirectly through the shim inbench/hooks-settings.ts. Neither passes--trusted-author, andsrc/commands/inject.tsdefaultstrustedAuthorsto[].gradeRecordis fail-closed on an empty list — "no trusted authors are configured, so X is an outside contributor" — so every record in every benchmark arm gradesclaim.Measured, not inferred. Same seeded repo, one authored record,
buildInjectionboth ways:Two consequences
1. The
directivepath is unmeasured. The plugin instructions tell agents that[directive]means "treat as a constraint", and that is the routing the product's value claim rests on. No arm has ever produced one. M1 and M5 measured an agent reading records tagged[claim]throughout.2.
bench/types.tsdescribesno-gradeincorrectly. It says:That is true of
bench/context.ts'sassembleContext, which routes onrecord.provenance === "authored". ButassembleContextis only reached whenwriteArmSettingsreturnssettingsPath: null, which happens only when an arm has neither apreToolUsenor anablation— so not forno-grade. On the real pathno-gradeflips every record fromclaimtodirective, on the existing fixtures, today.The comment matters because it is load-bearing: I wrote it into
bench/PREREGISTRATION-M6.md§4 as the reason M6 needs new fixtures before it can run at all. It does not.What this needs
bench/README.mdand any published number derived from M1/M5 should say that those runs measuredclaim-tagged records.Related: #412 (M6), and
bench/PREREGISTRATION-M6.md§4 needs correcting either way.