Add mix mob.mutate — break the code on purpose and check the tests notice (MOB-153) - #64
Merged
Merged
Conversation
…tice A green suite says the tests ran, not that they guard anything. This has been hand-rolled here roughly seven times and found real defects every time: a `record(...)` deletable from a frame tracker with 1500 tests green, an `-export` line whose removal fails load_nif and crash-loops every app at boot, a `requested:` option whose deletion silently restored the bug it was added to fix. It also got written wrong once, comparing each mutant against a hardcoded pass-count that was off by one, so every surviving mutation was reported as caught — including a genuine gap. The baseline is now always captured by running, and a baseline run that prints no `Result:` line refuses to continue rather than treating "no output" as a pass. An adversarial review found three blockers, all of which it demonstrated by running rather than by reading. The heredoc scanner never closed on `\"\"\")` — the form this task's own `Mix.raise(\"\"\"…\"\"\")` uses — so it stayed inside the heredoc to end of file and classified 185 of the task module's 226 lines as prose. Every line of the file-rewriting and restore logic, the riskiest code here, was invisible to the tool while the decision record claimed "we ran it on itself". Worse than the extent is the direction: a desync SKIPS code, so those mutants are never generated, never run and never reported, and the output is indistinguishable from a clean bill of health. Opening and closing are now separate tests, and the attribute state has a line budget so a stray bracket in a doc string cannot eat a module either. The docs promised a snapshot that restored the tree "even if the run is interrupted". There was no snapshot — the review sent SIGINT mid-run and left the file corrupted — and the sentence talked the reader out of the one habit that would have saved them. The run now refuses to start unless the files it would touch are clean in git, which is the only guarantee that survives being killed outright, and it also closes the case where an editor autosaves mid-mutant and `restore!` overwrites the edit while verifying successfully against its own stale snapshot. There was no extension filter, so a branch touching CHANGELOG.md or mix.lock — as this one does — produced a wall of guaranteed survivors, failed, and rewrote mix.lock line by line while running the suite against it. Source extensions are now a whitelist. Two more were this module's own thesis turned back on it. `classify/3` defaulted a mutant it could not measure to `killed`, which is the same score inflation the hardcoded baseline produced, and now reports `:unmeasured`. And `from_diff/1` swallowed git's exit status, so a typo'd `--base` or a missing origin/master measured nothing and reported `"outcome": "ok"` — the defect this repo fixed in `mix mob.deploy` one release earlier. It also concatenated two diffs with different line-number origins, and missed staged work entirely, so on a fully staged branch the default mode found nothing and called it success; both are now one `git diff HEAD`. Also fixes a mint CVE the security scanner surfaced (1.9.3 → 1.10.0, lock-only). The two remaining HIGH findings are the known OTP-tarball drift, pre-existing on master. Refs MOB-153 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GenericJam
force-pushed
the
feat/mob-153-mutate
branch
from
September 5, 2026 06:30
02ebdc6 to
e71c245
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A green suite says the tests ran, not that they guard anything. This has been hand-rolled here roughly seven times and found real defects every time — a
record(...)deletable from a frame tracker with 1500 tests green, an-exportline whose removal failsload_nifand crash-loops every app at boot, arequested:option whose deletion silently restored the bug it was added to fix.It also got written wrong once: comparing each mutant against a hardcoded pass-count that was off by one, so every surviving mutation was reported as caught, including a genuine gap. The baseline is now always captured by running, and a baseline that prints no
Result:line refuses to continue rather than treating "no output" as a pass.What the review found — all three demonstrated by running, not reading
The heredoc scanner never closed on
""")— the form this task's ownMix.raise("""…""")uses — so it stayed inside the heredoc to EOF and classified 185 of the task module's 226 lines as prose. Every line of the file-rewriting and restore logic was invisible to the tool while the decision record claimed "we ran it on itself".Worse than the extent is the direction: a desync skips code, so those mutants are never generated, never run, never reported — the output is indistinguishable from a clean bill of health. Opening and closing are now separate tests, and the attribute state has a line budget so a stray bracket in a doc string can't eat a module either.
The docs promised a snapshot that didn't exist. They said the tree was restored "even if the run is interrupted"; the reviewer sent SIGINT mid-run and left the file corrupted. The sentence actively talked the reader out of the one habit that would have saved them. The run now refuses to start unless the target files are clean in git — the only guarantee that survives being killed outright, and it also closes the case where an editor autosaves mid-mutant and
restore!overwrites the edit while verifying successfully against its own stale snapshot.No extension filter, so a branch touching
CHANGELOG.mdormix.lock— as this one does — produced a wall of guaranteed survivors, failed, and rewrotemix.lockline by line while running the suite against it.Two more were this module's own thesis turned back on it:
classify/3defaulted an unmeasurable mutant tokilled(now:unmeasured), andfrom_diff/1swallowed git's exit status so a typo'd--basemeasured nothing and reported"outcome": "ok"— the defect this repo fixed inmix mob.deployone release ago. It also mixed two diffs with different line-number origins and missed staged work entirely; both are now onegit diff HEAD.Dogfooding
Building it found four defects in itself before the review found five more: prose mutation, spec mutation, the heredoc desync, and six untested branches in its own diff parser. Running it on the task module now reports most of
run/1as surviving, which is accurate — this repo's convention is that a Mix task stays a thin unstubbed I/O wrapper with the decisions extracted and tested, sosource_file?/1,summary/4,dirty_message/2and all ofMobDev.Mutateare guarded and the orchestration is not. The moduledoc says so rather than letting the report read as a failure.Also fixes a
mintCVE the scanner surfaced (1.9.3 → 1.10.0, lock-only). The two remaining HIGH findings are the known OTP-tarball drift, verified pre-existing on master.2304 tests, credo and format clean, decision record included.