Skip to content

fix(mutmut): dot the last scripts/ test-loader gap, make pip_constraint_sets.py mutation-testable (#262) - #280

Open
cdeust wants to merge 2 commits into
mainfrom
fix-mutmut-attribution-gap-262
Open

fix(mutmut): dot the last scripts/ test-loader gap, make pip_constraint_sets.py mutation-testable (#262)#280
cdeust wants to merge 2 commits into
mainfrom
fix-mutmut-attribution-gap-262

Conversation

@cdeust

@cdeust cdeust commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

tests_py/scripts/test_typecheck_env_parity.py still loaded
scripts/pip_constraint_sets.py under the bare synthetic name
"pip_constraint_sets" instead of the path-derived "scripts.pip_constraint_sets"
mutmut keys mutant trampolines on. #264 fixed this exact defect for 14
sibling scripts/ test files but missed this one (found by code-reviewer
post-merge on #264, filed conceptually under issue #262).

Dotting the loader name alone was not sufficient to produce a real
mutant tally: ConstraintSet's three methods (path/command/header_lines)
carried 100% of the file's logic, and mutmut's mutation generator
categorically excludes the body of any @dataclass-decorated class (it
must — copying a decorated class for the trampoline setup can re-run the
decorator's side effects). Confirmed empirically: 0 mutants for the
whole file
with the logic as methods, vs 58 mutants after extracting
constraint_path / constraint_command / constraint_header_lines into
free functions and leaving ConstraintSet as pure data. Every call site
across generate_pip_constraints.py and its two test files is updated to
match; the pre-existing 301-test tests_py/scripts/ suite passes
unchanged, proving the refactor is behavior-preserving.

What changed

  1. tests_py/scripts/test_typecheck_env_parity.py — dotted the
    pip_constraint_sets loader (the reported gap).
  2. scripts/pip_constraint_sets.py — extracted ConstraintSet's three
    methods to module-level functions (data/behavior split), so the logic is
    no longer hidden inside a @dataclass-excluded class body.
  3. scripts/generate_pip_constraints.py + tests_py/scripts/test_generate_pip_constraints.py
    — updated every call site (.path()constraint_path(...), .command()
    constraint_command(...), .header_lines()constraint_header_lines(...)).
  4. tests_py/scripts/test_pip_constraint_sets.py (new) — a dedicated,
    direct unit-test file for the three extracted functions. Neither existing
    test file called them for real: every prior real invocation went through
    compose()/render(), which every existing test either mocks or
    short-circuits before reaching (this is what "0 killed, 31 no-tests, 22
    survived" surfaced on the first honest run, before this file existed).
  5. tests_py/scripts/test_generate_repo_badges.py — boy-scout find from
    the directory-wide sweep: it bare-imported scripts/badge_render.py
    (import badge_render after inserting scripts/ onto sys.path)
    instead of loading it under "scripts.badge_render" — same defect class.
    Fixed with its own dotted load, independent of generate_repo_badges.py's
    own (unaffected, still-bare) internal import of the same file.
  6. pyproject.toml.github was missing from [tool.mutmut] also_copy;
    test_typecheck_env_parity.py reads .github/workflows/ci.yml directly,
    so any scoped mutation run selecting it failed collection under every
    mutant with FileNotFoundError instead of scoring the suite. Found while
    producing the tally below; purely additive, no effect on any other
    scoped run (json_native.py/launcher_deps.py/generate_pip_constraints.py
    all read files already listed).
  7. Two ruff E501 violations this diff's own line growth introduced,
    fixed in the same commit (§14).
  8. Suite-count resync: 6550 → 6560 (+10 for the new test file) across the
    six hard-coded sites + the committed badge SVG, regenerated via
    generate_repo_badges.py --test-count 6560 and verified with
    check_doc_claims.py --test-count 6560.

Directory-wide sweep (deliverable #4)

grep -rn "spec_from_file_location\|sys.path.insert\|sys.path.append" tests_py/
— 21 call sites verified by hand:

  • 15 spec_from_file_location sites: all 13 already-fixed scripts/ test
    files confirmed still dotted correctly; the 1 reported gap (fixed above);
    1 more found and fixed (badge_render, above). tests_py/fuzz/test_corpus_replay.py
    and tests_py/infrastructure/test_config.py also use this pattern but
    load files outside the current committed mutmut source_paths = ["mcp_server"] scope and outside issue Mutation testing cannot attribute mutants for any scripts/ module (synthetic module names in tests_py/scripts) #262's scripts/ blast radius —
    test_config.py's reload is a deliberate one-off isolation instance
    alongside ~40 normal dotted importers of mcp_server.infrastructure.config
    that already provide correct attribution, so it is not this defect.
  • 6 sys.path.insert/append sites: 3 wiki-migration test files already use
    the correct from scripts.X import ... dotted form (citing issue Mutation testing cannot attribute mutants for any scripts/ module (synthetic module names in tests_py/scripts) #262 in
    their own comments); the other 3 (test_goal_maintenance.py,
    test_goal_maintenance_wiring.py, benchmark_harness.py) insert "."
    only to make mcp_server importable and already import it dotted
    (from mcp_server.core import ...) — not an instance of this defect.

Mutation tally (real, quoted — deliverable #2/#3)

Reproducing the reported abort, literal invocation exactly as this
task named the files:

$ scripts/mutation_check.sh tests_py/scripts/test_typecheck_env_parity.py scripts/pip_constraint_sets.py

(against the pre-fix tree, bare loader name) —

Stopping early, because we could not find any test case for any mutant.

After dotting the loader name alone (methods still on ConstraintSet,
before the data/behavior split): mutate_file_contents() on the file
returns 0 mutants — confirmed via direct call, independent of any
test — because mutmut's MutationVisitor._skip_node_and_children
categorically excludes @dataclass-decorated ClassDef bodies.

After the full fix, same literal 2-arg invocation (scripts/mutation_check.sh tests_py/scripts/test_typecheck_env_parity.py scripts/pip_constraint_sets.py,
scoping the test selection to exactly the ONE file this task named — the
script's CLI takes a single test path):

58/58  🎉 5 🫥 31  ⏰ 0  🤔 0  🙁 22  🔇 0  🧙 0
>>> survivors (must be empty, or documented equivalents):
    scripts.pip_constraint_sets.x_constraint_command__mutmut_2: survived
    ... (22 total, all in constraint_command)
    scripts.pip_constraint_sets.x_constraint_header_lines__mutmut_1..30: no tests

Real attribution now happens (no more abort) — but that ONE test file
alone never called constraint_command/constraint_header_lines with
enough assertions to kill everything, because pip_constraint_sets.py is
also exercised by test_generate_pip_constraints.py and the new dedicated
test_pip_constraint_sets.py, neither of which the script's single-test-path
CLI can express alongside the other in one invocation.

Full, honest triage — scoping pytest_add_cli_args_test_selection to
BOTH test_typecheck_env_parity.py and the new test_pip_constraint_sets.py
(mirroring scripts/mutation_check.sh's own pyproject.toml patching
logic by hand, then invoking mutmut run directly against the shared venv
to avoid uv run's fresh-venv side effect):

58/58  🎉 58 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0  🧙 0

58 mutants, 58 killed, 0 survived, 0 no-tests. Every one of the 53
non-trivially-killed mutants was a missing test, now added in
test_pip_constraint_sets.py — none were equivalent; every mutation
observed was a real, distinguishable behavior change (a wrong CLI flag, a
dropped accumulator, a corrupted banner line).

Boy-scout defect found, evidenced, NOT fixed (too large for this PR)

generate_repo_badges.py's RepoBadge class has the identical
@dataclass-exclusion gap: confirmed empirically, 234 total mutants for
the file, 0 attributed to RepoBadge.path()/.render(). Fixing it
requires the same shape of work as this PR (extract to free functions,
update every call site in generate_repo_badges.py + its test file, add
direct tests, re-run mutation testing) — a second, independent root-cause
fix of comparable size. Mixing it into this PR would violate §15.1 ("don't
mix the enabling refactor with the fix") and roughly double this diff's
blast radius beyond its named subject (pip_constraint_sets.py / issue
#262). Per this task's explicit no-new-issues instruction, reported here
with evidence rather than filed.

Test plan

  • pytest tests_py/scripts/ — 311 passed, 117 subtests passed
    (was 301 before this PR's new test file)
  • Full suite: pytest -q6560 passed, 117 subtests passed (167.83s)
  • python scripts/generate_pip_constraints.py --checkrequirements OK (13 checked)
  • python scripts/check_doc_claims.py --test-count 6560 — OK
  • python scripts/generate_repo_badges.py --check --test-count 6560 — OK
  • ruff check / ruff format --check on every touched file — clean
  • scripts/mutation_check.sh run twice (before/after) + a manually-scoped
    full-triage run — real tallies quoted above, 0 survivors in the final state
  • Production bare-importers of both refactored modules
    (generate_pip_constraints.py's from pip_constraint_sets import ...,
    generate_repo_badges.py / refresh_mcp_toplist_badge.py's
    import badge_render) verified to still resolve via real subprocess
    invocation, unaffected by the test-loader renames.

🤖 Generated with Claude Code

https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u

cdeust and others added 2 commits July 30, 2026 05:02
…raint_sets.py mutation-testable (#262)

test_typecheck_env_parity.py loaded scripts/pip_constraint_sets.py under
the bare name "pip_constraint_sets" instead of the path-derived
"scripts.pip_constraint_sets" mutmut keys mutant trampolines on — #264
fixed this for 14 sibling scripts/ test files but missed this one, so a
scoped mutation run on it aborted with "Stopping early ... none match
any mutant key" (found by code-reviewer post-merge on #264).

Dotting the loader alone was not sufficient: ConstraintSet's three
methods (path/command/header_lines) carried its entire logic, and
mutmut's mutation generator categorically excludes the body of any
@dataclass-decorated class (confirmed empirically: 0 mutants for the
whole file with the methods in place, vs 58 after extracting them to
free functions constraint_path/constraint_command/constraint_header_lines
with ConstraintSet left as pure data). Every call site across
generate_pip_constraints.py and its two test files is updated to match;
the existing 301-test tests_py/scripts/ suite passes unchanged, proving
the refactor is behavior-preserving.

A dedicated tests_py/scripts/test_pip_constraint_sets.py directly
exercises the three extracted functions (neither existing test file
called them for real: everything went through compose()/render(),
which every test either mocks or short-circuits before reaching). The
scoped mutation run (tests_py/scripts/test_typecheck_env_parity.py +
test_pip_constraint_sets.py against scripts/pip_constraint_sets.py) now
produces real attribution: 58 mutants, 58 killed, 0 survived, 0 no-tests.

Boy-scout finds in the same sweep (grepped every spec_from_file_location
and sys.path-plus-bare-import in tests_py/, 21 call sites verified):
- tests_py/scripts/test_generate_repo_badges.py bare-imported
  scripts/badge_render.py (`import badge_render` after a sys.path
  insert) instead of loading it under "scripts.badge_render" — same
  defect class, now fixed with its own dotted load, independent of
  generate_repo_badges.py's own bare import of the same file.
- Two ruff E501 violations introduced by this diff's own line-length
  growth, fixed in the same commit (§14).
- generate_repo_badges.py's RepoBadge class has the identical
  @dataclass-exclusion gap (confirmed: 234 mutants total, 0 attributed
  to RepoBadge.path()/.render()) — out of this PR's blast radius (a
  separate root-cause fix of comparable size; mixing it in here would
  violate §15.1's "don't mix the enabling refactor with the fix");
  reported with evidence rather than filed, per this task's no-new-
  issues instruction.

Resynced the 6 suite-count sites + badge SVG to the measured absolute
(6560, +10 for test_pip_constraint_sets.py) via generate_repo_badges.py
and manual doc edits; both check_doc_claims.py --test-count 6560 and
generate_repo_badges.py --check --test-count 6560 pass. Full suite:
6560 passed, 117 subtests passed (167.83s).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u
…onstraint_sets.py scoped runs

tests_py/scripts/test_typecheck_env_parity.py reads .github/workflows/ci.yml
directly (cross-checking CONTRIBUTING.md's documented type-check command
against it). Running scripts/mutation_check.sh with this test in the
selection failed collection under every mutant with FileNotFoundError
(mutants/.github/workflows/ci.yml missing) instead of scoring the suite --
found while producing the real mutant tally for the previous commit's
pip_constraint_sets.py fix. .github wasn't in also_copy for any prior
scoped run (json_native.py, launcher_deps.py, generate_pip_constraints.py
all read files already listed), so this is additive and does not change
behavior for any other scoped run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Yu6EnWspTfqHoGkExyS6u
@cdeust

cdeust commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Discharging the boy-scout deferral flagged in review.

The RepoBadge / generate_repo_badges.py @dataclass mutation-exclusion gap (234 mutants for the file, 0 attributed to .path()/.render()) is being fixed now in its own PR, not filed as an issue and not folded in here — folding it in would mix two independent root-cause fixes (§15.1), which is why the reviewer was right that it does not belong in this PR.

Branch: fix-repobadge-mutation-attribution. It applies the same treatment this PR established for ConstraintSet, and is required to triage every mutant the change exposes rather than just reporting a number.

That is a stronger discharge of §14 than an issue number would be: the defect gets fixed, not tracked. Also noted from review, non-blocking: the sweep tally in this PR's body should read 22 sites (15 + 7), not 21 (15 + 6) — tests_py/scripts/test_generate_repo_badges.py:25's sys.path.insert was uncounted. The reviewer confirmed it is benign and that no further site exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant