Skip to content

Order-dependent failure: test_benchmark_db_resolves_with_psycopg_present fails under random test ordering #402

Description

@cdeust

Symptom

tests_py/benchmarks/test_lib_init_no_psycopg.py::test_benchmark_db_resolves_with_psycopg_present
fails during a full-suite run, but only under some orderings produced by
pytest-randomly.

Evidence (measured 2026-08-09, macOS, uv run --no-sync, branch fix/issue-400-agent-briefing-fallback)

Run Command Result
Full suite, random order pytest -q 1 failed, 7383 passed, 1 skipped — the failure above
Full suite, deterministic order pytest -q -p no:randomly 7243 passed, 142 skipped, 0 failed
The module alone pytest tests_py/benchmarks/test_lib_init_no_psycopg.py -q 4 passed
The whole directory, 3 consecutive runs pytest tests_py/benchmarks/ -q 58 passed each time

So the polluter is outside tests_py/benchmarks/, and the failure is
invisible in deterministic order.

Why this matters

The module under test manipulates import state (psycopg presence /
sys.modules), which is exactly the shared state a random ordering exposes.
Per ~/.claude/rules/coding-standards.md §6.2, a flaky test is a symptom of
hidden shared state, not noise to retry away. Left alone, it will fail CI at
random and train everyone to re-run the job.

Not yet established

Which test pollutes the import state. Reproducing needs the pytest-randomly
seed from a failing run plus a bisect over the ordering; that was not done
here because the finding is outside the blast radius of the change that
surfaced it (issue #400, which touches mcp_server/hooks/agent_briefing*.py
only). This issue exists so it is not lost.

Suggested first step

Capture the seed line (Using --randomly-seed=...) from a failing full-suite
run, then pytest -p randomly --randomly-seed=<seed> and bisect the ordering
to name the polluter.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions