Skip to content

No benchmark arm configures a trusted author, so the [directive] path the README's claims rest on has never been measured #415

Description

@MongLong0214

What I found

commitlore-on runs the real injector (commitlore inject --hook-input), and the ablation arms run buildInjection directly through the shim in bench/hooks-settings.ts. Neither passes --trusted-author, and src/commands/inject.ts defaults trustedAuthors to [].

gradeRecord is fail-closed on an empty list — "no trusted authors are configured, so X is an outside contributor" — so every record in every benchmark arm grades claim.

Measured, not inferred. Same seeded repo, one authored record, buildInjection both ways:

--- commitlore-on (real grading, no trustedAuthors) ---
Ruled-out
  [claim]      r-probe01  a4613ed5  shared Redis cache | ops refuses another stateful dependency

--- no-grade ---
Ruled-out
  [directive]  r-probe01  a4613ed5  shared Redis cache | ops refuses another stateful dependency

Two consequences

1. The directive path is unmeasured. The plugin instructions tell agents that [directive] means "treat as a constraint", and that is the routing the product's value claim rests on. No arm has ever produced one. M1 and M5 measured an agent reading records tagged [claim] throughout.

2. bench/types.ts describes no-grade incorrectly. It says:

Inert on a task whose records are all Provenance: authored, because there is nothing to promote.

That is true of bench/context.ts's assembleContext, which routes on record.provenance === "authored". But assembleContext is only reached when writeArmSettings returns settingsPath: null, which happens only when an arm has neither a preToolUse nor an ablation — so not for no-grade. On the real path no-grade flips every record from claim to directive, on the existing fixtures, today.

The comment matters because it is load-bearing: I wrote it into bench/PREREGISTRATION-M6.md §4 as the reason M6 needs new fixtures before it can run at all. It does not.

What this needs

  • Decide whether the benchmark should configure a trusted author. It is a design choice, not an oversight to patch quietly: an arm with one measures the product as a repository actually configures it, and an arm without one measures the fail-closed default. Both are defensible and they answer different questions.
  • Whichever is chosen, record it in the arm definition so a reader of a result knows which was measured.
  • bench/README.md and any published number derived from M1/M5 should say that those runs measured claim-tagged records.

Related: #412 (M6), and bench/PREREGISTRATION-M6.md §4 needs correcting either way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions