A golden repo that exercises all 16 domains needs a third expectation state first

[rodlunt]

Raised while looking at whether the three-agent comparison run could use the eval harness for a scored, apples-to-apples leg. It can, but only over three domains, and extending that turns out to be blocked by the spec format rather than by the writing.

Where the golden repo is today

evals/golden/repo/ is GrindPoints: 8 files, 234 lines. expected.json declares "domains": ["d01", "d05", "d16"] and holds 11 expectations, 7 plants and 4 controls. That is the taster pack's three domains, 54 of 260 rules.

So a scored eval currently answers "did the agent find the planted defects and stay quiet on the controls" for three domains, and says nothing about the other thirteen.

What a 16-domain golden repo would have to be

Not more code. The preconditions the other thirteen domains presume are mostly satisfiable by an artefact in the tree, which means the repo stops being eight files of code and becomes a small system plus its paperwork:

Domain	What has to exist	Example plant
d03	a diagram under docs/	an activity diagram with a fork and no join
d08	docs/risk-assessment.md	an asset register with no owners; an ALE that does not beat its control's cost
d09	docs/incident-response.md	RTO plus WRT exceeding MTD; a response role with no named alternate
d11	docker-compose.yml, a cache, a consumer	a cache with no invalidation story; no dead-letter queue
d13	docs/estimate.md	a flat 20 per cent contingency; a figure carrying no basis or date
d14	docs/postmortems/	a write-up that adopted the reporter's theory and never got a reproduction
d06, d07, d10, d15	workflows, handlers, templates	tag-pinned actions, f-string SQL, 200-on-error, placeholder used as a label

That shape works. It also changes what roughly half the eval measures, from finding a bug in code to reading a document critically. Those are different skills and the set should be deliberate about the split rather than discovering it afterwards.

The blocker

Expectation.expect is Literal["finding", "no-finding"] (evals.py:102, and again on ExpectationOutcome at evals.py:184). Two states.

The scoring is strict about the second, and correctly so: a no-finding control counts as held only when the rule is verdicted pass. Not-applicable, could-not-evaluate and no verdict at all every one score control-not-evaluated, with the reason written into the code at evals.py:474-484: "The control never actually ran, so it cannot be counted as having held."

That is right for what it does today. It also means there is no way to write "this rule's precondition does not hold in this repository, and not-applicable is the correct answer".

On a small fictional service that is the correct answer for a large share of 260 rules. It is also the exact behaviour #100 and #178 exist to enforce, and the single most valuable thing a 16-domain eval could measure. As the spec stands, an auditor that gets those rules right scores control-not-evaluated, which reads as a non-result rather than as a pass.

The first step

A third expectation state, expect: "not-applicable", held when the rule is verdicted not-applicable and carries a note. The note is not a detail: #100 made it mandatory precisely because a not-applicable without one was the cheap escape hatch, so an expectation that accepts a bare not-applicable would re-open the hole it is meant to close.

Both Literals move together, and a fourth outcome value is needed alongside held, false-positive and control-not-evaluated for the case where a rule expected not-applicable came back as pass (the auditor claimed to have checked something that does not apply, which is its own defect and deserves its own name).

Then extend, do not rewrite

Add one domain at a time to GrindPoints, its artefacts and its expectations together, rather than authoring a second repo. expected.json's domains list is already the scoping mechanism and the scorer handles a run that audited extra domains cleanly, gathering their findings into unexpected_findings from every recorded domain rather than only the spec's.

One consequence to plan for: the domain list is a contract. _check_structural_gates raises EvalStructuralError when a spec domain has no result in the run, or has one whose status is not completed (evals.py:346-355). So the day d06 joins the list, every eval run has to audit d06. That is correct and it is strict, and it means the list should grow only as fast as people are willing to pay for the runs.

What it costs

Today: 234 lines, 11 expectations. A thin 16-domain version at two plants, one control and two not-applicable cases per domain is roughly 80 expectations across perhaps 40 files.

The maintenance side is the real cost. The pack revises monthly and an expectation is pinned to a rule's wording, so a bigger eval set decays faster. That debt already exists at small scale: the pack's own revision logs record d06-R15 and the late d05 rules as having no golden example yet.

What it still would not tell you

GrindPoints is fiction. At 234 lines or at 2,000 it sets a floor rather than predicting behaviour on a real 344-file codebase. And evals/README.md already records the harness's own limit: it scores whatever one run produced, with no majority vote across runs and no variance tracking, so a single flaky verdict moves the score.