Skip to content

Add mix mob.mutate — break the code on purpose and check the tests notice (MOB-153) - #64

Merged
GenericJam merged 1 commit into
masterfrom
feat/mob-153-mutate
Sep 5, 2026
Merged

Add mix mob.mutate — break the code on purpose and check the tests notice (MOB-153)#64
GenericJam merged 1 commit into
masterfrom
feat/mob-153-mutate

Conversation

@GenericJam

Copy link
Copy Markdown
Owner

A green suite says the tests ran, not that they guard anything. This has been hand-rolled here roughly seven times and found real defects every time — a record(...) deletable from a frame tracker with 1500 tests green, an -export line whose removal fails load_nif and crash-loops every app at boot, a requested: option whose deletion silently restored the bug it was added to fix.

It also got written wrong once: comparing each mutant against a hardcoded pass-count that was off by one, so every surviving mutation was reported as caught, including a genuine gap. The baseline is now always captured by running, and a baseline that prints no Result: line refuses to continue rather than treating "no output" as a pass.

What the review found — all three demonstrated by running, not reading

The heredoc scanner never closed on """) — the form this task's own Mix.raise("""…""") uses — so it stayed inside the heredoc to EOF and classified 185 of the task module's 226 lines as prose. Every line of the file-rewriting and restore logic was invisible to the tool while the decision record claimed "we ran it on itself".

Worse than the extent is the direction: a desync skips code, so those mutants are never generated, never run, never reported — the output is indistinguishable from a clean bill of health. Opening and closing are now separate tests, and the attribute state has a line budget so a stray bracket in a doc string can't eat a module either.

The docs promised a snapshot that didn't exist. They said the tree was restored "even if the run is interrupted"; the reviewer sent SIGINT mid-run and left the file corrupted. The sentence actively talked the reader out of the one habit that would have saved them. The run now refuses to start unless the target files are clean in git — the only guarantee that survives being killed outright, and it also closes the case where an editor autosaves mid-mutant and restore! overwrites the edit while verifying successfully against its own stale snapshot.

No extension filter, so a branch touching CHANGELOG.md or mix.lock — as this one does — produced a wall of guaranteed survivors, failed, and rewrote mix.lock line by line while running the suite against it.

Two more were this module's own thesis turned back on it: classify/3 defaulted an unmeasurable mutant to killed (now :unmeasured), and from_diff/1 swallowed git's exit status so a typo'd --base measured nothing and reported "outcome": "ok" — the defect this repo fixed in mix mob.deploy one release ago. It also mixed two diffs with different line-number origins and missed staged work entirely; both are now one git diff HEAD.

Dogfooding

Building it found four defects in itself before the review found five more: prose mutation, spec mutation, the heredoc desync, and six untested branches in its own diff parser. Running it on the task module now reports most of run/1 as surviving, which is accurate — this repo's convention is that a Mix task stays a thin unstubbed I/O wrapper with the decisions extracted and tested, so source_file?/1, summary/4, dirty_message/2 and all of MobDev.Mutate are guarded and the orchestration is not. The moduledoc says so rather than letting the report read as a failure.

Also fixes a mint CVE the scanner surfaced (1.9.3 → 1.10.0, lock-only). The two remaining HIGH findings are the known OTP-tarball drift, verified pre-existing on master.

2304 tests, credo and format clean, decision record included.

…tice

A green suite says the tests ran, not that they guard anything. This has been
hand-rolled here roughly seven times and found real defects every time: a
`record(...)` deletable from a frame tracker with 1500 tests green, an
`-export` line whose removal fails load_nif and crash-loops every app at boot,
a `requested:` option whose deletion silently restored the bug it was added to
fix. It also got written wrong once, comparing each mutant against a hardcoded
pass-count that was off by one, so every surviving mutation was reported as
caught — including a genuine gap. The baseline is now always captured by
running, and a baseline run that prints no `Result:` line refuses to continue
rather than treating "no output" as a pass.

An adversarial review found three blockers, all of which it demonstrated by
running rather than by reading.

The heredoc scanner never closed on `\"\"\")` — the form this task's own
`Mix.raise(\"\"\"…\"\"\")` uses — so it stayed inside the heredoc to end of file
and classified 185 of the task module's 226 lines as prose. Every line of the
file-rewriting and restore logic, the riskiest code here, was invisible to the
tool while the decision record claimed "we ran it on itself". Worse than the
extent is the direction: a desync SKIPS code, so those mutants are never
generated, never run and never reported, and the output is indistinguishable
from a clean bill of health. Opening and closing are now separate tests, and
the attribute state has a line budget so a stray bracket in a doc string cannot
eat a module either.

The docs promised a snapshot that restored the tree "even if the run is
interrupted". There was no snapshot — the review sent SIGINT mid-run and left
the file corrupted — and the sentence talked the reader out of the one habit
that would have saved them. The run now refuses to start unless the files it
would touch are clean in git, which is the only guarantee that survives being
killed outright, and it also closes the case where an editor autosaves
mid-mutant and `restore!` overwrites the edit while verifying successfully
against its own stale snapshot.

There was no extension filter, so a branch touching CHANGELOG.md or mix.lock —
as this one does — produced a wall of guaranteed survivors, failed, and
rewrote mix.lock line by line while running the suite against it. Source
extensions are now a whitelist.

Two more were this module's own thesis turned back on it. `classify/3`
defaulted a mutant it could not measure to `killed`, which is the same score
inflation the hardcoded baseline produced, and now reports `:unmeasured`. And
`from_diff/1` swallowed git's exit status, so a typo'd `--base` or a missing
origin/master measured nothing and reported `"outcome": "ok"` — the defect
this repo fixed in `mix mob.deploy` one release earlier. It also concatenated
two diffs with different line-number origins, and missed staged work entirely,
so on a fully staged branch the default mode found nothing and called it
success; both are now one `git diff HEAD`.

Also fixes a mint CVE the security scanner surfaced (1.9.3 → 1.10.0,
lock-only). The two remaining HIGH findings are the known OTP-tarball drift,
pre-existing on master.

Refs MOB-153

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@GenericJam
GenericJam merged commit 76c4d3c into master Sep 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant