ADR 0156 — ASVS scorecard as data: derived count, verified evidence anchors, fail-closed drift gate

[wshallwshall]

The ASVS score is maintained as prose and nothing checks it. One re-anchoring session on 2026-08-01 produced, in a single day:

Observed	Count
Re-derivations of the headline count from scratch	6
Residuals of record factually false at HEAD	12
Of those, absence claims that had silently stopped being true	5
Cells missing from an enumeration called "arithmetic-checked and complete"	10
Documents asserting a superseded count as current	5
Subagent tokens spent verifying	~23 million

Most of that wasn't discovery. It was re-deriving facts already known but never durably recorded, then re-verifying them because the recorded version had gone quietly false.

The instructive one is the dropped ten: it survived because the arithmetic closed to 345 and closure was read as proof. Closure only proves the four buckets sum — not that every cell landed in one. No self-consistent count can detect an absent cell.

What this adds

scripts/asvs/scorecard.py — stdlib only (tomllib/json/re), no new dependency, and deliberately data-free: it takes the scorecard and the ASVS corpus as paths, so this repo unit-tests it against fixtures while the vault runs the same code against real posture data.

Check	The defect it kills
Every corpus id appears exactly once, nothing outside it appears	the 10 dropped cells
Count computed, never typed	5 documents asserting 3 counts
Each cell's evidence names a token that must still resolve	the 12 false residuals
An absence claim needs a positive control that must still match	the 5 false absence claims
unverified is a first-class verdict	~219 inherited Passes hiding inside a headline
Missing scorecard → exit 2, never 0	ASVS 15.1.3

On absence specifically: a grep naming the wrong token returns zero and reads exactly like proof. That's how five residuals stayed false for weeks. So the schema stores the search, not just its conclusion — and if the positive control goes quiet, the claim is void regardless of what the pattern returns.

Tests

17 tests, and every check is proved to go RED before it is trusted green — dropped cell, duplicate cell, an id ASVS retired in 5.0, a level disagreeing with the corpus, a moved token, a deleted file, a blind absence search, an absence that became false.

Exit codes verified by hand: 0 clean · 1 findings · 2 could-not-measure.

ruff check / ruff format --check / mypy all clean.

§7 was amended at ratification — the premise was false

The ADR originally proposed "a vault CI job runs the verifier against the real data." Open question 1 asked whether that CI actually executes. It does not — every vault workflow is disabled_manually, last run 2026-07-27, and both vault PRs that day merged with zero checks.

A CI-only design would have shipped dead. Built instead as a vault pre-commit hook plus one narrow new workflow (new workflows are active by default even though the estate is off).

That §7 was a confident, plausible, unchecked claim about system state is this ADR's own thesis applied to itself. It was caught only because ratification was gated on answering the open question rather than on the argument reading well.

What this does not do

It does not make the score correct — only consistent, derived and drift-detecting. A wrong verdict recorded carefully is still wrong; adversarial verification remains the only cure for that, and this ADR doesn't replace it.

The companion vault PR carries the real scorecard and the enforcement.

---

🤖 Generated with Claude Code