A measurement-smell linter — fail a PR that changes retrieval with no interval

[akash-coded]

Half-formed, posting it here rather than as an issue because it does not have a hypothesis yet.

The PR template asks for a measurement table. CI checks that the eval gate passes. Neither
checks whether the claims in the PR description are supported by the numbers in it.

Things a linter could catch:

• A delta quoted without an interval
• An interval that straddles zero, described with the word "improves"
• A metric quoted at more decimal places than the noise band justifies
• A comparison across different question sets or slices, presented as a before/after
• The word "significant" without a p-value or an interval anywhere nearby

Most of those are regex-able against the PR body plus the eval JSON.

Where I am unsure: this could easily become annoying and get disabled, which is worse than
not having it. And a linter that produces false positives on honest PRs teaches people to
ignore CI comments generally.

Possibly the right shape is a comment that asks a question rather than a check that fails.
"This PR describes an improvement — I could not find an interval for it. Is that in the eval
output?"

Does anyone think this is worth building, or is it solving a problem the template already
solves well enough?

[akash-coded]

Dana · reviewer · owns the eval harness

In favour, with a scope reduction that I think is the difference between this shipping and this becoming a maintenance burden.

Do not try to detect bad statistics. A linter that tries to judge whether an interval was computed correctly will be wrong constantly and people will disable it within a fortnight.

Do detect missing statistics. That is a syntactic check and it is nearly always right:

• PR touches nanorag/retrieve.py, metrics.py, chunking.py, embed.py or context.py
• and the body contains no line matching CI [ or inside the noise band or verdict
• → fail, with a message pointing at the PR template

That is twenty lines and it catches the actual failure mode, which is not "someone computed a bad interval" but "someone shipped a retrieval change with a point estimate and no interval at all". The needs: eval-numbers label already encodes the same intent manually; this just stops it being manual.

[akash-coded]

Priya · cohort peer

Agreed, and one addition that costs nothing: make it a warning for the first month, then a failure. A new gate that blocks merges on day one gets bypassed by someone with admin rights and then never re-enabled.

[akash-coded]

Both accepted. Scoping this to the syntactic check, warn-then-fail, and opening it as an issue against P7 · Hardening with the twenty-line spec in the body rather than leaving it as an idea. Thanks — the "detect missing, not bad" line is what made it buildable.

[akash-coded]

Filed as #56, against P7 · Hardening, with the twenty-line spec in the body.

Both refinements from this thread made it in verbatim:

• Detect missing, not bad. The issue is explicit that parsing the interval or judging whether the metric moved is out of scope. If someone writes CI [made up, numbers] the check passes, and that is correct — it is a completeness check, not a referee.
• Warn for a month, then fail. Written into the acceptance criteria rather than left as an intention, because that is the difference between a control and theatre.

Dependabot PRs are exempted; they cannot produce a measurement.