Symptom
tests_py/benchmarks/test_lib_init_no_psycopg.py::test_benchmark_db_resolves_with_psycopg_present
fails during a full-suite run, but only under some orderings produced by
pytest-randomly.
Evidence (measured 2026-08-09, macOS, uv run --no-sync, branch fix/issue-400-agent-briefing-fallback)
| Run |
Command |
Result |
| Full suite, random order |
pytest -q |
1 failed, 7383 passed, 1 skipped — the failure above |
| Full suite, deterministic order |
pytest -q -p no:randomly |
7243 passed, 142 skipped, 0 failed |
| The module alone |
pytest tests_py/benchmarks/test_lib_init_no_psycopg.py -q |
4 passed |
| The whole directory, 3 consecutive runs |
pytest tests_py/benchmarks/ -q |
58 passed each time |
So the polluter is outside tests_py/benchmarks/, and the failure is
invisible in deterministic order.
Why this matters
The module under test manipulates import state (psycopg presence /
sys.modules), which is exactly the shared state a random ordering exposes.
Per ~/.claude/rules/coding-standards.md §6.2, a flaky test is a symptom of
hidden shared state, not noise to retry away. Left alone, it will fail CI at
random and train everyone to re-run the job.
Not yet established
Which test pollutes the import state. Reproducing needs the pytest-randomly
seed from a failing run plus a bisect over the ordering; that was not done
here because the finding is outside the blast radius of the change that
surfaced it (issue #400, which touches mcp_server/hooks/agent_briefing*.py
only). This issue exists so it is not lost.
Suggested first step
Capture the seed line (Using --randomly-seed=...) from a failing full-suite
run, then pytest -p randomly --randomly-seed=<seed> and bisect the ordering
to name the polluter.
Symptom
tests_py/benchmarks/test_lib_init_no_psycopg.py::test_benchmark_db_resolves_with_psycopg_presentfails during a full-suite run, but only under some orderings produced by
pytest-randomly.Evidence (measured 2026-08-09, macOS,
uv run --no-sync, branchfix/issue-400-agent-briefing-fallback)pytest -q1 failed, 7383 passed, 1 skipped— the failure abovepytest -q -p no:randomly7243 passed, 142 skipped, 0 failedpytest tests_py/benchmarks/test_lib_init_no_psycopg.py -q4 passedpytest tests_py/benchmarks/ -q58 passedeach timeSo the polluter is outside
tests_py/benchmarks/, and the failure isinvisible in deterministic order.
Why this matters
The module under test manipulates import state (
psycopgpresence /sys.modules), which is exactly the shared state a random ordering exposes.Per
~/.claude/rules/coding-standards.md§6.2, a flaky test is a symptom ofhidden shared state, not noise to retry away. Left alone, it will fail CI at
random and train everyone to re-run the job.
Not yet established
Which test pollutes the import state. Reproducing needs the
pytest-randomlyseed from a failing run plus a bisect over the ordering; that was not done
here because the finding is outside the blast radius of the change that
surfaced it (issue #400, which touches
mcp_server/hooks/agent_briefing*.pyonly). This issue exists so it is not lost.
Suggested first step
Capture the seed line (
Using --randomly-seed=...) from a failing full-suiterun, then
pytest -p randomly --randomly-seed=<seed>and bisect the orderingto name the polluter.