measurement: does the suite notice when a conformance check breaks

[imran-siddique]

Refs #53. Authored by @lywinged, from their branch lywinged/trace-tests:measurement, rebased onto current main. Their commit and authorship are intact. Opened as a maintainer because the branch had no PR.

What it measures. Not whether the checks are right, they are. Whether a regression inside a conformance module would be caught. That distinction matters more here than in ordinary code: this repository's output is a conformance stamp, so a broken check does not produce a wrong answer, it produces a missing question, and implementations keep getting stamped without the property being checked.

mutate_modules.py rewrites one Finding(..., Status.FAIL, ...) site at a time so it can never fail, runs the suite, restores, and verifies the restore before continuing. It refuses to start on a non-green baseline, refuses a rewrite that did not change the file, and refuses an empty site list.

I reproduced the numbers on current main rather than taking them from the report:

checks measured : 18
verified        : 15
unverified      :  3  ['TR-SIG-002', 'TR-TXN-001', 'TR-TXN-002']
margin 1 only   :  7
sites           : 23 of 33 guarded, 10 unguarded

All three TR-SIG-002 and TR-TXN-001 sites are unguarded. TR-TXN-001 is the only place the Level 2 tool-transcript requirement is enforced, so today that requirement could stop being checked and every test would still pass.

Why mutation rather than reading the tests, which is the part worth keeping: grepping test files for TR-xxx-nnn codes and diffing against emissions reports 12 unguarded checks, mutation reports 3. Tests exercise modules through check(...) and assert on findings without naming codes, so naming is a convention and changing outcome is the property. It cuts the other way too: five files mention tool_transcript, which reads as coverage, but one re-implements the digest check inline, two assert on the JSON schema, and one only asserts the module ran.

Scope. This lands the instrument and its report under measurement/, not the ten missing tests and not a CI gate. Wiring it into CI is a separate decision: it takes about a minute to run, and it should probably ratchet rather than block on first failure.

🤖 Generated with Claude Code

[lywinged]

Thank you for opening it, and the stale-install finding is a real hole rather than an environment quirk. It belongs in the same list as the three guards the docstring already claims, and it is the one that would have embarrassed the measurement rather than merely broken it.

The three existing guards all defend against a mutation that did not take effect: a red baseline, a rewrite that changed nothing, an empty site list. Yours is the same failure one layer down — the mutation takes effect on a tree the suite never imports. The symptom is identical to the strongest possible result, every site reporting that nothing noticed, and 0 of 18 verified is exactly what a suite with no guarded checks at all would print. An instrument that cannot separate I measured nothing from there is nothing to measure has to refuse rather than report.

The guard, in the same shape as the others, checked before the baseline rather than inferred after:

def assert_suite_imports_this_checkout() -> None:
    """The suite must import the tree being mutated, not another copy of it.

    The guards above defend against a mutation that did not take effect. This one
    defends against it taking effect on a tree the suite never imports: an editable
    install pointing elsewhere, a wheel earlier on sys.path, a stale .pth. The symptom
    is indistinguishable from a perfect result at the other end, so it is a precondition
    and not a diagnosis.
    """
    probe = subprocess.run(
        [sys.executable, "-c",
         "import pathlib, trace_tests; print(pathlib.Path(trace_tests.__file__).resolve())"],
        cwd=TRACE_TESTS, capture_output=True, text=True,
    )
    if probe.returncode != 0:
        sys.exit(
            "could not import trace_tests with the interpreter that runs the suite:\n"
            + probe.stderr.strip()
        )
    imported = Path(probe.stdout.strip())
    expected = (TRACE_TESTS / "src" / "trace_tests").resolve()
    if not (expected == imported.parent or expected in imported.parents):
        sys.exit(
            f"the suite imports trace_tests from {imported.parent}\n"
            f"but this run mutates            {expected}\n"
            "Every site would report that nothing noticed, which is also what a suite "
            "with no guarded checks looks like. Reinstall with `pip install -e .` from "
            "this checkout, or run with PYTHONPATH=src."
        )

It uses sys.executable and cwd=TRACE_TESTS deliberately, so the probe resolves the import the same way run_suite() does rather than the way this script happens to have been launched. I have tested the path comparison against a correct checkout, a submodule inside it, and an install elsewhere; I have not been able to run it against a real stale install, so if your reproduction is easy to recreate it is worth pointing at it once.

enum_drift.py has the same exposure and no equivalent check, so the guard should be called from both rather than living in one.

On delivery: contrib/regression-guard-measurement is on this repository and I have read access only, so I cannot push to it. Say which you prefer and I will do that one — you take the patch above into your branch, or I open a small separate PR against main after this merges. I am not opening one unasked while your branch is live, since that would only give you a conflict to resolve.