mutation testing

Mutation testing

Coverage says a line ran. Mutation testing asks whether anything would have noticed if it were wrong — it corrupts the source one edit at a time and reports which corruptions the suite fails to catch. For a project whose entire value is defensible numbers, that is the more useful question, and it has now found real gaps twice (see findings.md).

Running it

.venv/bin/python -m pip install -e ".[mut]"
.venv/bin/mutmut run

On demand, never in CI. A full sweep is orders of magnitude slower than the suite, and a surviving mutant is a prompt to think rather than a build failure.

Useful subsets and follow-ups:

.venv/bin/mutmut run "gratinglab.geometry.*"   # one module
.venv/bin/mutmut results                        # survivors, by name
.venv/bin/mutmut show gratinglab.geometry.x_beta__mutmut_13   # the diff
.venv/bin/mutmut browse                         # interactive

Note the mutant-name prefix is gratinglab., not src.gratinglab. — the src layout does not appear in the names.

What is mutated, and what is not

[tool.mutmut] in pyproject.toml is the authority. In summary:

Excluded	Why
gui/	Computes nothing. A surviving mutant there means a misplaced button, not a wrong number — which is the whole reason that layer was arranged to be thin.
io/	Pinned by round-trip and corpus-parsing tests, which a mutation operator does not reach usefully.
illumination.py, problem.py, profiles.py	A tool limitation, not a judgement. See below.

Everything else — geometry, checks, convergence, compare, result, solvers/ — is mutated. That is where the physics is.

The pydantic exclusion

mutmut rewrites every function into a trampoline plus a class attribute holding the mutant table. On a BaseModel subclass, pydantic rejects the un-annotated attribute outright and the module fails to import — so the choice is not "mutate them badly", it is "mutate them or import them". Their validators are covered by tests/test_illumination.py and tests/test_profiles.py.

If this matters more later, the fix is upstream (model_config['ignored_types'] would have to be set on every model, which is production code changed for a test tool — not worth it today).

Two things that make this work at all

Both are load-bearing and easy to break by accident.

pythonpath = ["src"] in [tool.pytest.ini_options]. mutmut copies the project to mutants/ and runs pytest there. Without this, import gratinglab resolves to the real source via the editable install, every mutant "survives", and the run reports a catastrophe that never happened. It is a no-op for an ordinary pytest, where it names the same path the editable install already provides.

No tests/__init__.py. With one, pytest imports conftest as tests.conftest, which cannot resolve when pytest is driven in-process from a console script — the current directory is not on sys.path there. The package layout was not buying anything: the one thing it enabled, a relative from .conftest import reference_dir, is better served by tests/corpus.py, since conftest.py is not an ordinary module and importing from it is fragile by construction.

Version pin

mut = ["mutmut>=3.2,<3.3"], deliberately not in dev.

mutmut 3.3+ requires libcst, which ships no macOS x86_64 wheel and must be compiled with a Rust toolchain — mutmut's own README names this case ("known for at least the x86_64-darwin architecture"). 3.2.3 uses parso and is pure Python. Lift the pin when either libcst ships the wheel or the machine grows a Rust toolchain; nothing else depends on it.

Where it stands

796 mutants across the physics core. The first full sweep killed 763 — 95.9% — with 18 of the 33 survivors in check_reciprocity alone. Those are now closed, so checks.py and geometry.py both sit at 100% (geometry's two remaining survivors are equivalent mutants, verified as such). The breakdown and what closing them took are in findings.md.

The useful lesson from that round: a strategy with no trace on its own output cannot be tested through the public API. check_reciprocity picks which orders to test and the report never says which — so verifying it needed a recording stand-in solver, not a cleverer assertion against the real one.

A full sweep takes roughly half an hour on this machine (~6 mutants/second), which is why it is a deliberate act rather than a CI step.

Reading a survivor

Not every survivor is a missing test. An equivalent mutant changes the source without changing behaviour, and no test can kill it. Two of the four survivors in geometry.beta were equivalent, verified by evaluating both versions rather than by assuming:

• |s| <= 1.0 → |s| <= 2.0: arcsin of anything outside [-1, 1] is NaN regardless, so the guard is doing less than it looks.
• np.asarray(s, dtype=np.float64) → np.asarray(s): NumPy promotes anyway.

So the workflow is: read the diff, work out whether the change is observable at all, and only then decide whether it is a missing test. Record the equivalent ones — in findings.md — so the next reader does not re-derive them.

A third category: observable, but only where the model is already void

M16-D left one survivor that is neither a missing test nor strictly equivalent. In _local_reflected_efficiency, replacing $\sqrt{r_i}\sqrt{r_d}$ with $\sqrt{r_i r_d}$ does change the output — but only where the branch wrap is non-uniform across the groove, which is the Brewster region near normal incidence on a deep groove. Everywhere else the two differ by a global sign that cancels out of $|\int\cdots|^2$, and on Au at 1–6 nm they agree to $10^{-15}$.

Killing it would mean asserting on orders carrying $10^{-9}$, in a regime the solver already reports as outside what the model can carry. That is pinning noise to make a counter go up. It is left alive deliberately, and the reasoning lives in the _local_reflected_efficiency docstring and in test_the_two_square_roots_are_taken_separately, which asserts the property that makes it harmless rather than the arbitrary choice between the two forms.

Worth naming as its own category, because "survivor ⇒ missing test" would have produced a bad test here, and "survivor ⇒ equivalent mutant" would have been a false claim.