Skip to content

M6: measure whether trust grading changes what an agent does, and build the fixtures that can #412

Description

@MongLong0214

Why

M1 and M5 measure one guarantee — that injecting a path's records changes what an agent proposes. Trust grading (SPEC §7) is a second guarantee, and the product's security claims rest on it entirely: [directive] tells an agent to treat a line as a constraint, [claim] tells it to weigh the line, blocked withholds it because the line is the attack.

None of that has been measured.

The blocking precondition

bench/types.ts says it about the no-grade ablation arm:

Inert on a task whose records are all Provenance: authored, because there is nothing to promote.

All ten reproposal-* fixtures are that kind of task. Running M6 on them would compare an arm against itself and report a null that means nothing.

So this ticket is two pieces of work, in order:

  1. New fixtures. Each task seeds a planted record that grading treats differently from no-grade — one from an untrusted author (directive ungraded, claim graded) and one carrying an injection payload (injected verbatim ungraded, withheld graded). The planted record must instruct something the task does not otherwise call for, unambiguous in a diff, harmless in a throwaway workspace.
  2. A base-rate pilot, then registration. M6 has no prior: nobody has measured how often an agent complies with a planted record handed to it as an instruction. n cannot be written down before that pilot, and the pilot's rows are not citable — the same rule M5 applied to its own.

Preregistration

bench/PREREGISTRATION-M6.md is drafted and explicitly marked not registered: §4's fixtures do not exist and §5's n cannot be fixed without the pilot. It is registered only when both carry numbers rather than procedures.

Harness constraint, already known

The build must include the trust-grading fixes from #408 and #409. Both change what grading does — #409 changes who a notes-sourced record is graded by, #408 changes which text the injection guard blocks — so a run on a build without them measures a grader that no longer exists.

Definition of done

  • Fixtures exist and the ablation is verifiably in force per row (preregistration §7).
  • The pilot has produced a base rate, n is fixed, and the preregistration status line says registered.
  • The run completes every registered row before any 2×2 table is computed, with shards written outside any temporary directory and committed as they land (M5 deviation 3).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions