Repository navigation
v1.1.0 #1584
Replies: 1 comment
|
Field report from running 1.1.0 on a real project, alongside mutant as a control. Setup: a private Rails application — Rails 8.1.2, Ruby 4.0.1, RSpec, SQLite. evilution 1.1.0, mutant 0.16.3 run second as a comparison baseline, over four months and eight side-by-side runs. Everything below was re-verified against 1.1.0 with Two of these are outright bugs with short reproductions, so they come first. 1. A multi-file run hides a file that has no spec at all (bug)Alone, a file with no resolvable spec is reported honestly: Add one well-tested file to the same command and it vanishes from the verdict:
This is not hypothetical — we shipped a service with no spec whatsoever because the run it was part of headlined Agents are especially prone to this: the headline is what gets read and reported back. Suggestion: a file named on the command line that resolves to no spec is a different condition from an individual 2. The default
|
| Run | Total | Verified | Neutral | Printed |
|---|---|---|---|---|
4 controllers, -j 4 |
361 | 65 | 289 | score 1.0 / PASS |
same, -j 1 |
361 | 264 | 90 | score 1.0 / PASS |
3 services, -j 1 |
536 | 251 | 282 | score 1.0 / PASS |
The first row reports full marks on 18% of the run. Running serially converted 224 of those 289 into kills, which confirms they were infrastructure rather than semantics — but we only found that out by re-running by hand.
Asks: attach a cause to every neutral (baseline_failure / no_targeted_examples / infra_error(<class>)); print the verified denominator next to the score (score 1.00 (251 of 536 verified, 282 neutral)); warn or auto-retry serially when the ratio is high; and for the no_targeted_examples cause specifically, prefer the full-file fallback over recording neutral.
5. Two missing operators, both of which found real dead code
Block unwrap. block_removal deletes the block with its body:
$ bundle exec evilution util mutation -e 'def a; Base.transaction { x = mk; sp(x); x }; end'
7. block_removal — #a (line 1)
+ def a; Base.transaction; end
The method now does nothing, so anything kills it. What's missing is keeping the body and dropping the wrapper (+ def a; x = mk; sp(x); x; end). For a transaction that's the only mutation that asks whether atomicity is actually tested — the happy path is identical either way. It found three things for us: one genuinely untested transaction, and two transactions that were dead code, because the bodies used non-bang update/destroy which never raise, so nothing could ever roll back. That second case is the interesting one: the mutation didn't reveal a missing test, it revealed decorative code. Worth applying to transaction, tap, with_lock, benchmark, Timeout.timeout, instrument — probably not to blocks whose value feeds a collection.
Per-key hash removal. hash_literal only empties the whole hash ({} or nil), both loud and trivially killed. Dropping one key at a time ({ x: s, y: nil } → { x: s }) is what exposed two defensive keys of ours that model validations made unreachable. Option and attribute hashes are everywhere in Rails, and a permanently-redundant key is a normal way for them to rot.
Minor: is_a? → instance_of? isn't generated. The two differ only for subclasses, so it survives whenever the receiver is never subclassed — which is the signal that the type check is broader than needed. For us it prompted deleting two type switches in favour of a method on the two classes.
6. --format json isn't valid JSON when coverage is on
Without NO_COVERAGE=1, SimpleCov's report is appended after the JSON object, so JSON.parse fails:
85.00%).
Lowest-coverage files (line):
0.00% app/controllers/some_controller.rb
SimpleCov failed with exit 2 due to a coverage related error
Consumers have to set NO_COVERAGE=1 (not always wanted) or scan for the first { and hope. Machine-readable output on stdout and everything else on stderr would fix it; an --output FILE flag would sidestep it. Related: SimpleCov's non-zero exit also leaks into the run's exit status, which interacts badly with using the exit code as a gate.
Where 1.1.0 is clearly ahead
Reporting the other side fairly, since the comparison ran both ways:
- Per-worker SQLite isolation is decisive. mutant has no equivalent, so on this project it's single-job only —
-j 2invents survivors that vary run to run (4–5 alive on a service where-j 1reports 0). A controller sweep costs ~35 minutes against evilution's ~5. - Neutral classification on request specs is more truthful. Where mutant reports a phantom
Neutral failurefor every controller subject — reproducible on unchanged code — evilution correctly classifies the same situations and gives a truthful score. - Richer operator set on literals and collections, and
index_to_fetchin particular caught a real nil-handling bug for us that mutant's light set doesn't generate.
A correction to something I reported earlier
I'd previously written that evilution lacks a "condition to constant" operator, on the strength of a mutant if true survivor that our evilution run scored 100% on. That was wrong: conditional_negation already emits both if true and if false. So evilution generated that mutation and didn't surface it as a survivor — which makes it a reporting question (item 3/4 above), not an operator gap. Withdrawn rather than filed.
Happy to split any of this into separate issues with fuller reproductions if that's easier to triage — items 1 and 2 in particular are self-contained and I have the exact commands.
Uh oh!
There was an error while loading. Please reload this page.
What's Changed
Full Changelog: v1.0.0...v1.1.0
This discussion was created from the release v1.1.0.
All reactions