Skip to content

fix(measurement): purge __pycache__, so a run measures the tree it just rewrote - #60

Open
lywinged wants to merge 2 commits into
agentrust-io:mainfrom
lywinged:fix/measurement-harness-determinism
Open

fix(measurement): purge __pycache__, so a run measures the tree it just rewrote#60
lywinged wants to merge 2 commits into
agentrust-io:mainfrom
lywinged:fix/measurement-harness-determinism

Conversation

@lywinged

@lywinged lywinged commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Two commits. #59 took the import-guard half of this branch; what remains is the cache purge, plus tests for the guards themselves — which nothing was checking, including the one #59 landed.


1. Purge __pycache__, so a run measures the tree it just rewrote

Status.FAIL and Status.PASS are the same length, so rewriting a site leaves the file size unchanged and a run can end up measuring the previous iteration's bytecode. The failures that iteration caused are then attributed to the site currently under mutation: margins come out larger than they are, and a check held by exactly one test is reported as comfortably covered.

Two fresh clones, one at 424c2f8 and one at this commit, each run starting from a purged cache, nothing else touching either directory:

424c2f8       margin-1 watch list:  0, 0, 0, 2
this commit   margin-1 watch list:  8, 8, 8, 8

The list has eight entries — REPORT.md names them and calls them "the number to watch". On main that watch list came out empty three times in four. That is the wrong direction to be wrong in: the run reports a healthier suite than the one that exists.

The second symptom needs no numbers. After four runs on 424c2f8:

$ pytest -q
14 failed, 147 passed, 5 xpassed

$ git status --porcelain -- src/
                                  # nothing

Every mutated source file was restored, and the restoration is verified. The interpreter was still loading bytecode compiled from a mutated one. REPORT.md said the checkout was "restored and re-verified green after every mutation" — true of the files, not of what Python then executed. Corrected here.

2. Tests for the guards

This came out of an audit: revert each fix landed in these two repositories and count what fails. Ten of eleven have a test that notices. The one that does not is #59 — delete assert_suite_imports_this_checkout() from main() and the suite stays green at 161 passed.

measurement/ had no tests at all, so every guard in the script was in exactly the state the script exists to detect.

Thirteen tests, one broken precondition each, against a synthetic checkout — the smallest tree the script accepts, carrying one guarded check and one unguarded one so the reporting path is covered and not only the aborts.

Load-bearing, measured rather than claimed. Removing each guard and re-running:

Guard removed Tests that fail
checkout not found 3
no FAIL sites 2
refusing to guess the line 1
import path (#59) 2
green baseline 1
__pycache__ purge 2
restore after each site 3

Every one of the thirteen fails under at least one removal. The purge accounts for two, one of them the reporting control — the synthetic tree reproduces the cache contamination rather than only asserting the directory is gone.

Two things this turned up while writing it

A test named for the "refusing to guess" guard was reaching "no FAIL sites" instead: its fixture bound Status.FAIL to a name, so the walk never collected the site. Renamed, given a fixture that reaches the guard, and the original behaviour kept as its own test — a Finding whose status arrives through a name is not collected, and the run reports on what it found without mentioning the omission. Nothing in this repository writes them that way and the site count is checkable against grep -c Status.FAIL, so it is recorded rather than fixed, and the test fails if that stops being true.

Run as a subprocess rather than imported, because the script resolves TRACE_TESTS at module scope and importing it would bind that to this repository. Exit codes are not asserted alone: a refusal exits 1, and so does a completed run that found an unverified check.


Also in this diff

Verification

Fresh clone, in a directory that has never held the repository: 174 pass, 5 xpassed, identical across two consecutive runs. ruff clean on the new file, and the same 68 pre-existing errors either side of this change.

The A/B measurement above was re-run from scratch after an earlier attempt was invalidated — I had run an unrelated git checkout inside the directory the experiment was using, which swapped the script under test mid-run. Given what this PR is about, that seemed worth saying rather than quietly re-reporting.

@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@imran-siddique

Copy link
Copy Markdown
Member

The cache purge is the important half of this and I want it. Eight runs on one unchanged tree returning 8, 5, 2, 0, 5, 2, 5, 5 is a harness that cannot be cited, and the reason is exactly as you say: Status.FAIL and Status.PASS are the same length, so a rewrite leaves the file size untouched and the interpreter is free to load the previous iteration's bytecode.

The part that matters most is your point that it fails upward. A guard that inflates margins reports unguarded checks as verified, and this repo's whole output is a conformance claim, so that is the one direction it must never fail in. The REPORT.md note about restoration is the same insight and worth keeping: git status clean while pytest fails against stale bytecode is a genuinely confusing state to leave behind.

One thing to change. I merged #59 a few minutes ago, which adds the same import guard as assert_suite_imports_this_checkout. You wrote the two independently and they collide, so this branch now conflicts.

Could you rebase onto main and drop verify_import_path? That leaves _purge_caches and the REPORT.md updates, which is the right scope. Keep your REPORT.md wording on the cache guard, it is better than what #59 carried, and the 161 passed correction should come along too.

One difference worth knowing: #59 tests expected == imported.parent or expected in imported.parents, where yours tests only the second. Both are right for a normal layout.

…st rewrote

agentrust-io#59 landed the import-path guard from this branch. This is the other half, and
it is the half that fails upward.

Status.FAIL and Status.PASS are the same length, so rewriting a site leaves the
file size unchanged and a run can end up measuring the previous iteration's
bytecode. The failures that iteration caused are then attributed to the site
currently under mutation. Margins come out larger than they are, and a check
held by exactly one test is reported as comfortably covered - which is the one
direction this instrument must never fail in.

Measured on two fresh clones, one at 424c2f8 and one at this commit, each run
starting from a purged cache, nothing else touching either directory:

    424c2f8      margin-1 watch list:  0, 0, 0, 2
    this commit  margin-1 watch list:  8, 8, 8, 8

The list has eight entries; REPORT.md names them and calls them the number to
watch. On main that watch list came out empty three times in four.

The second symptom is visible without reading any numbers. After four runs on
424c2f8, pytest reports 14 failures against a tree git status calls clean: the
source is restored, the bytecode is not. On this commit pytest is green after
every run.

REPORT.md is corrected on that point - it said the checkout was "restored and
re-verified green after every mutation", true of the files and not of what
Python then executed - and its guard list now covers both the guard agentrust-io#59 added
and this one. The module docstring says five guards and lists five; agentrust-io#59 added a
fourth without updating the count.

The expected baseline in the reproduction block is updated from 118 to 161,
which is what this tree produces.

Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lywinged
lywinged force-pushed the fix/measurement-harness-determinism branch from d8a9e48 to ddeffa2 Compare August 12, 2026 16:42
@lywinged lywinged changed the title fix(measurement): the harness disagrees with itself between runs fix(measurement): purge __pycache__, so a run measures the tree it just rewrote Aug 12, 2026
@lywinged

Copy link
Copy Markdown
Contributor Author

Force-pushed onto 424c2f8 and narrowed. #59 took the import-path guard, so this is now only the cache purge, and the title and description are updated to match.

Worth flagging one thing that changed with it: assert_suite_imports_this_checkout as merged is better than what this branch had, because it checks the probe's return code. The version here was silent if the interpreter could not import trace_tests at all. I dropped mine and kept yours rather than reconciling them.

The A/B numbers in the description are freshly measured on two clones at 424c2f8 and at this commit. They are not the numbers that were here before — the earlier run was taken on the pre-#59 tree, and a second attempt was invalidated when I ran an unrelated git checkout inside the directory the experiment was using, which swapped the script under test mid-run. Given what this PR is about, that seemed worth saying rather than quietly re-reporting.

The audit that prompted this: revert each fix we have landed and count what
fails. Ten of eleven have a test that notices. The one that does not is agentrust-io#59 -
delete assert_suite_imports_this_checkout() from main() and the suite stays
green at 161 passed. measurement/ had no tests at all, so every guard in the
script was in exactly the state the script exists to detect.

Thirteen tests, one broken precondition each, run against a synthetic checkout -
the smallest tree with the shape the script requires, carrying one guarded check
and one unguarded one so the reporting path is covered and not only the aborts.

Run as a subprocess rather than imported, because the script resolves
TRACE_TESTS at module scope and importing it would bind that to this repository.
Exit codes are not asserted alone: a refusal exits 1 and so does a completed run
that found an unverified check, so every test asserts on the message.

Load-bearing, measured rather than claimed. Removing each guard and re-running:

    checkout not found          3 tests fail
    no FAIL sites               2
    refusing to guess the line  1
    import path                 2
    green baseline              1
    __pycache__ purge           2
    restore after each site     3

Every one of the thirteen fails under at least one removal. Two are covered by
the purge alone, one of them the reporting control - the synthetic tree
reproduces the cache contamination rather than only asserting the directory is
gone.

Two things this turned up while writing it.

A test named for the "refusing to guess" guard was reaching "no FAIL sites"
instead: its fixture bound Status.FAIL to a name, so the walk never collected
the site. Renamed and given a fixture that reaches the guard, with the original
behaviour kept as its own test - a Finding whose status arrives through a name
is not collected, the run reports on what it found without mentioning the
omission, and nothing here writes them that way. Recorded rather than fixed, and
the test fails if that stops being true.

174 pass, and identical across two consecutive runs. ruff clean on the new file,
and the same 68 pre-existing errors either side of this change.

Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants