ditto v0.12.0

A score that no longer mixes two questions

ditto staged scoped 43 mutants to a staged change spanning five packages and judged them with one command naming one of them. 17 of the 20 survivors lived in a package that command never compiles, so nothing it ran could have killed them: the run's ceiling was 0.605, and the number printed as a verdict was 0.53 against a bar of 0.80. The score was measuring the tests and the command's scope together, and nothing in the output told the two apart.

A release now reads which packages the test command executes — from the command's own go test -json stream, which the baseline run already produces, resolved through go list -deps -test so that a mutant in a dependency of a named package is correctly judged rather than accused. The mutants it cannot compile:

• leave the numerator and the denominator, exactly as a mutant that never compiled does (S = D/(M − E)), so the printed score is a real measurement of what could be judged;
• are not run, because a guaranteed survivor bought with a full suite run is not evidence about anybody's tests;
• are named with the package that holds them, and the run exits 3.

┃ • Total:       26                    ┃
┃ • Killed:      23                    ┃
┃ • Survived:     3                    ┃
┃ • Unmeasured:       17               ┃
┃ ⨯ Score:     0.88 (minimum: 0.80)    ┃
┃ 17 of the 43 mutants in this scope are never compiled by this test command, so
┃ nothing it runs can kill them and they are out of the score entirely:
┃   internal/desktop (17)
┃ Name every package the scope mutates in --test-command, or narrow the scope.

Exit 3 is the other half: a scope ditto cannot measure and a score below the bar were both 1, and a gate has to be able to tell them apart, because no test answers the first — only naming every package the scope mutates, or narrowing the scope, does.

--include-prefix

The mirror of --exclude-prefix, on staged and changed. The fix above is what makes it necessary: a run refuses a scope its command cannot compile, and the honest answer is usually to narrow the run to the package the command names — one flag, where one exclusion per other package the change happens to touch does the same.

PlanStaged, RunStaged, PlanChanged and RunChanged take a ditto.Prefixes where they took one []string, so what a run is about and what it leaves out are one argument instead of two that can drift.

Verification

Checked on the tagged commit ac3c2ddce0310122a3d7d76862478ba5331c9351:

• CI, both matrix legs (Go 1.27 and latest): pass
• CodeQL, OSV-Scanner, SonarCloud: pass
• chained mutation gate over the change: 118 mutants, 94 killed, 24 survived, score 0.80 against a 0.50 minimum
• local gate: lint 0 issues, the full suite green with a cold cache (go test -count=1 ./...), and the exact counters passed with mutantsPerReleaseOnThisRepository at 948

The behaviour was read off the shipped binary, not off the suite: a throwaway git repository with three packages — one the command names, one its tests import, and one nothing compiles — run under a command that names only the first. It printed the block above, with internal/untested named and exit 3; the same fixture under ./... reported nothing unmeasured and exited 0. That fixture and its golden are now in the repository, with the control beside it, because the middle package is the case that decides whether the check is usable at all: a mutant in a package the command does not name is killable by that named package's tests.

What this release does not claim

• The check only speaks when the command reports itself. It reads the packages the command executes out of its own go test -json stream, so a --test-command that is make, gotestsum, a wrapper script or go test without -json gets no check at all, and keeps the behaviour it always had.
• It works per package, not per file. A file behind a build tag for another operating system is inside a package the command does compile, so its mutants stay ordinary survivors — which they are.
• A gated run is not checked, because Gated() replaces the plan of the exact module-scope command, whose packages the gated path already resolves for itself; asking the ordinary laboratory would buy a second suite run to learn what it already knows.
• Per-package test commands are not shipped. Grouping a scope's mutants by owning package and deriving one command per group is the version of this that would make the check unnecessary rather than enforceable, and it is a cost-model change: docs/backlog.md entry 29 records the measurement that would have to argue for it.
• One line of output moved deliberately: the baseline announcement prints before the first file's announcement, because asking the laboratory which packages the command compiles is what pays for the baseline. Every verdict and every mutant address is identical, and the golden says so where someone will look.

The whole release is one report's defect, committed with it: docs/reports/ditto-mutation-scope.md.

ditto v0.11.0

Gated() answers the complete module scope

Gated() compiled the mutated file's own package with go test -c, which is a narrower question than the default go test -count=1 ./.... A mutant in package P that only a test in a package importing P can kill survived it — and nothing in the output could show the difference, because the survivor addresses matched and the run was faster either way.

A gated run now discovers the module's package layout once, compiles the complete default scope's test binaries, and runs each of them for every selected mutant.

Gating replaces only the commands whose complete execution plan has been measured: go test -count=1 ./... and its built-in -json form in either flag order. Any custom or unsupported WithTestCommand — a package-local scope, an extra flag, extra spacing, an alternate executable — keeps the ordinary laboratory and the runner exactly as configured, counted as fallen back. The set is closed on purpose: a command that is almost the default is not the default.

A module whose test binaries collide now gates

The module-scope runner named every test binary after its package, so on a tree where two packages produced the same name — a root module, cmd/x and internal/x all build x.test — it refused before compiling anything, gating fell back for every file, and the repository gated nothing while looking like any other gated run. Measured on ditto itself: 0 of 850 mutants compiled.

Packages are now planned into batches free of name collisions — case-folded on Windows, where the filesystem is — and each batch compiles into its own directory. A collision-free module still plans exactly one batch, so the one-compilation win is unchanged.

Verification

Checked on the tagged commit 0d315b211b3e7ea56f704e44e1eee78057714075:

• CI, both matrix legs (Go 1.27 and latest): pass
• CodeQL, OSV-Scanner, SonarCloud: pass
• chained mutation gate over the change: 85 mutants, 74 killed, 11 survived, score 0.87 against a 0.50 minimum
• local gate: lint 0 issues, 522 tests passed, 15 exact counters passed

The first pull request was red on the Linux runner with validateBatches - goos always receives "linux" while a Windows host reported nothing: the parameter is the seam that makes Windows naming testable on a Linux host, and its only varying call sites were tests passing the literal "linux". Reproduced locally with GOOS=linux golangci-lint run, fixed by covering the Windows case-fold rather than by suppressing the finding, and re-verified at 0 issues under that target. Recorded in docs/learning-log.md.

What this release does not claim

The module path was measured, not assumed, and the follow-up work that did not earn its place is recorded as refutation:

• shared compilation plus fail-fast semantics removed 208 package starts from a real ten-mutant run and was still slower in every rotated round (1.1284, 1.0801, 1.0429)
• outer parallelism with an explicit two-worker scheduler overlapped exactly and preserved every verdict, but returned only 5.62% (ratio 0.943784 against a pre-registered 0.75 limit)
• the configured test command is 99.3% of a real staged run, so no precision-preserving headroom was measured inside ditto for that scope

Known open items, stated rather than implied as fixed: a module-path deadline kill is reported as an assertion; sources that are not gofmt-clean get no gating and are reported as none; and the heavy-suite ratio is unmeasured at this revision.

The Gated: none of 85 line in this repository's own CI gate is the admission rule working: that gate runs a configured command, not the default, so every mutant keeps its own path.

v0.10.0 — ditto -v, and a scope with nothing in it

Fixed

• A change with nothing mutable in it no longer fails ditto changed. The
  score calculator reports -1 for an empty run, which is below every threshold,
  so a scope that produced no mutants and a suite that failed were the same
  answer. Run and RunStaged keep that refusal — a repository or an index
  yielding nothing is a scope somebody configured wrong — but a diff is not: a
  new switch case, a comment, a rename are all legitimately unmutatable, and
  failing a gate for one is failing a change for being correct.

  Found by this repository's own gate the day after it went green, on the -v
  commit below: one added switch case, a scope of one file, zero mutants, and a
  red run that had found nothing wrong with anything.

  The count is read through an optional interface, and the reporter decorators
  now forward it. Without that it was unreadable exactly when the stack is
  deepest, which is the gated run that raised the problem — backlog entry 12's
  class, caught again.

Added

• -v and --version alongside ditto version. Nothing new is reported;
  it is the spelling people actually type, and a tool that only answers the long
  form makes them look it up to ask the simplest question it has.

  Safe at this level rather than lucky: subcommand flags are parsed by their own
  FlagSet, and Go's flag package does no prefix matching, so ditto run -v
  never resolved to -verbose and still does not — it reports an undefined flag,
  as it always did.

Measured

The gate on this release's merge: 9 mutants, score 1.00 against a 0.50 minimum, PASS in 66.19s.

It took four runs to get there, and each one found something the suite did not:
a scope of zero mutants read as a failing score; the -1 that means "unknown"
unguarded in three places; and then those guards written in the wrong package —
a gated mutant is judged by the tests of its OWN package, so assertions covering
exactly those lines passed while the mutants survived.

v0.9.0 — the gate finishes, and says what it measured

The release that makes ditto's own gate finish, by making it ask a question it
can afford.

Added

• ditto changed --since <ref>, and PlanChanged / RunChanged behind it.
  It is staged asked of a committed range instead of the index: the scope is
  <ref>...HEAD, the diff against their merge base, so a base that has moved on
  does not drag somebody else's commits into the bill. The diff parsing, the byte
  offsets, the fail-open rule and the sandbox are the staged path's, unchanged.

  It exists because ditto staged cannot be a CI gate. A CI checkout has nothing
  staged — the change is already committed — so a gate pointed at the staged
  scope skips, reports success, and measures nothing.

  It refuses a checkout with staged changes in it. A range scope names bytes of
  HEAD while the sandbox is written from the index, and those agree exactly
  while the index agrees with HEAD; scoping against one and mutating the other is
  the defect already measured at seven of eight verdicts moving. A worktree-only
  modification never reaches the sandbox and is allowed — written the other way
  round first, it refused ditto's own CI in fifty-one seconds, because the Devbox
  install step modifies a tracked lockfile.

  There is no default base, and there will not be one: on a CI checkout the
  useful base is the last release, on a branch it is the trunk, and a base
  guessed wrong is either a bill nobody asked for or a scope of nothing reported
  as green.

Changed

• ditto's own CI gate mutates the change rather than the repository. The
  repository-sized run asks for 783 mutants and dies at its thirty minutes having
  reached about 424, measured four times. Both levers were already spent: gating
  removes 54% of the compilations and does not close it, and cutting the mutant's
  suite by 46% moved the gate by 0.5%, because -failfast already stops a killed
  mutant at its first failing test. The bill was the wrong size rather than badly
  paid — backlog entry 21, open since the measurement and now closed.

  make test.mutation is untouched and workflow_dispatch still reaches it. The
  repository-sized question is worth asking on purpose; it was being asked on
  every push, against a clock it could not beat.

Measured

This repository's own gate, before and after:

Run	Result
Repository-sized, on the 0.8.0 merge	FAIL … 1800.181s — the -timeout=30m exactly
Changed gate, on this release's merge	PASS in 153.96s — 49 mutants, score 0.90 against a 0.50 minimum

v0.8.0 — what a run says while it is running

A release about what a run SAYS. Every item came from one exchange with a
consumer repository whose ditto staged run printed nothing for over ten
minutes, twice, and was reported as a hang. It was not stuck. Nothing it did was
wrong; nothing it did was visible either, and that turned out to be the defect.

Added

• A progress line per mutant. A release names each file and its mutant count
  before running any of them, and names each mutant before handing it over —
  before, not after, because a line printed once a mutant has finished cannot say
  which mutant a stall is inside.

  This is the whole of the reported problem. ditto printed nothing at all between
  Releasing Ditto… and the report, so a run advancing normally and a run that
  was genuinely stuck produced identical bytes. "No mutants appeared after ten
  minutes" was evidence of nothing, which is why it was misread as a setup stall
  and then misdiagnosed a second time as a baseline that had hit its deadline.
  The reporter's own measurement — a 27.4s suite — already refuted that, and
  nobody multiplied it by the mutant count because neither number was on screen.

• The baseline says what the suite cost. The laboratory already ran the suite
  once on unmutated code and threw the duration away. It is the per-mutant price
  of the test command, and printed beside the mutant count it is the whole bill.

• ditto version, which also answers --version and -version. The number
  comes from runtime/debug rather than a constant, so there is no second place
  for it to go stale, and a binary built from a checkout says it has no released
  version instead of naming one.

  It is not cosmetic: whether a mutant that never compiled is scored as killed
  differs between 0.6.0 and 0.7.0, and until now nothing a user could type said
  which build they had.

• --gated on ditto staged. run always had it. Nothing about one
  compilation per file is specific to a whole-repository release, and the staged
  path mutates fewer files, which is where a compilation is easiest to repay.

• --confirm-kills, opt-in on both subcommands. The baseline check is a
  sync.Once: it refuses a suite that is ALREADY red, and cannot see one that
  goes red at mutant 37, where a spurious failure becomes a kill no test earned
  and nothing in the report tells it from a real one.

  Only assertion kills are re-run — a mutant that never built already leaves the
  score on both sides, and a deadline is a clock ditto fired itself. A survivor
  is never re-run either: a flake manufactures failures, and cannot make a mutant
  the tests caught look like it escaped. Off by default, because it doubles the
  price of every assertion kill and buys nothing on a suite that does not flake.

Fixed

• --test-command rendered as -test-command -json in -h, which reads
  like a second flag. flag.PrintDefaults takes the first backquoted word in a
  usage string as the flag's VALUE NAME. Found by running the binary; no
  assertion on the help constant could have seen it, and the new test renders the
  flag set instead.

Changed

• --test-command's description names its cost. It said what the flag does
  and what its default is, and a reader who understood every word still ran
  ./... against a 27-second suite — once per mutant, sequentially. It now says
  that, and says what dropping -json costs rather than only what it does.

  The readme carries the same warning with the measured delta, because two
  separate repositories wrote that sentence into their own docs rather than
  getting it from here.

v0.7.0 — what ditto's answers are worth

A release about ditto's own answers: whether they can be believed, and whether the gate that produces them ever finishes. Both were measured rather than assumed, and both were wrong.

Breaking

The mutation score changes value for the same code. A mutant that never compiled used to be counted as killed — a failing compile exits non-zero, and that is how ditto recognises a kill — and it now leaves the numerator and the denominator.

Measured on a fixture with two non-viable mutants of five: 0.60 → 0.33. One earned kill of one viable mutant, which is what the tests actually did. If a WithMinimumThreshold was tuned against an older score, expect it to move — toward the truth.

On this repository, 83 of 748 mutants do not build: 11.1%, against a benchmark of Major 1.8% and PIT 0%.

The default test command gains -json. It is what lets ditto say why a mutant died.

The headline

A verdict now carries its reason. Ditto read any non-zero exit as a killed mutant — a failed assertion, a mutant that never compiled, a timeout, a missing binary. Measured on one file, 78 mutants: 22% of the kills were not assertions.

And ditto had no deadline anywhere. A test binary invoked directly takes -test.timeout 0 — disabled — so on the gated path a mutant that loops never returned. Both paths are bounded now, and a mutant the deadline stops is a kill carrying its own reason, as PIT, Stryker and Infection all do.

make test.mutation.staged is a gate that measures the change and finishes: 4 mutants, 82 seconds, green, against 424 of 727 that die at thirty minutes. Not a smaller timeout — a smaller question.

Which checks ran on the tagged commit, and which did not

Green by push event on bbef658: Continuous Integration (both legs, 1.27 and latest), CodeQL, OSV-Scanner, SonarCloud.

Red: the Mutation Tests workflow. It times out at 1800s having reached 424 of 727 mutants, and it has failed on every push since 2026-08-27 — 0.4.0, 0.5.0 and 0.6.0 all shipped over it. That is stated here rather than left to be discovered.

It is not a regression and it is not a mystery. It is measured three times, with both levers spent: gating removes 54% of the compilations and does not close it, and cutting the mutant's suite by 46% moved the gate by 0.5%, because -failfast already stops a killed mutant at its first failing test. The bill is the wrong size rather than badly paid, and docs/backlog.md entry 21 carries it.

Full notes in CHANGELOG.md, and the measurements behind each claim in docs/experiments/.

v0.6.0 — a repository can name what git does not carry

A sandbox is built from the index, and that stays. It is what makes a verdict about the change rather than about the desk it was written on — measured at 7 of 8 verdicts moving when a release read the working tree instead.

But some repositories do not build from their index alone. A generated directory the build embeds, generated bindings: on disk, not in git. The sandbox arrives without them, the package that needs them cannot compile, and the red-baseline guard refuses. Correct, and useless.

Added

.ditto.json, at the repository root:

{"generated": ["frontend/dist", "frontend/wailsjs"]}

Those paths are copied from the working tree after the index is materialised.

The guards are the design, not decoration:

• Naming a path git tracks is refused. A tracked path has an index version, and that is the one a staged run measures. Letting the working tree win there is exactly the hole the sandbox exists to close.
• A named path that is not there is refused, not skipped. It was named because the build needs it, so a sandbox quietly missing it produces the failure this setting was added to prevent.
• Every copy is announced. Those are the only bytes in the sandbox that did not come from git, and a reader deciding whether to trust a verdict deserves to know which ones.

Both refusals were watched refusing before being trusted.

Measured, with its control

On a repository whose root package cannot compile from its index:

	result
without .ditto.json	refuses — pattern all:frontend/dist: no matching files found
with it	9 mutants, 8 killed, 1 survived, 0.89

The same verdicts a tracked placeholder produced, and now no placeholder is needed — which matters beyond tidiness, because a tracked file inside a build-output directory is rewritten by every build. docs/experiments/a-placeholder-in-build-output.md prices that and the two alternatives, and declines to choose between them.

Also

internal/filecopy replaces two byte-identical copies of the same function. SonarCloud caught real duplication rather than a style opinion, and both callers need it for the same reason: a sandbox is only a sandbox if what it holds is a copy.

Unchanged, and measured to be

The eight exact counters in perf/baseline.json, and both goldens byte-identical. 363 tests.

What ran

Tagged on de4f233, the exact merged head, after a green run on it: CI (1.27) and CI (latest) both pass on ubuntu-latest, with CodeQL, Analyze (go), OSV-Scanner and SonarCloud.

v0.5.0 — a sandbox is a copy, not a reference

Two entries, one defect seen from two sides. Both were found by measuring rather than by reading, and the second one corrects the first fix that shipped in this branch.

Fixed

A sandbox mirrored the repository as symlinks. That is a reference to the tree, not a copy of it, and the two part company in two ways that matter.

It fails loudly on reads. Go refuses to embed an irregular file, so no package with a go:embed directive could build in a sandbox at all — every release before this one silently could not measure them. Found on a real repository where it reported 9 mutants, 9 killed, a perfect 1.00 for a package that never compiled, because a failing command is how ditto recognises a killed mutant. The truth is 8 killed and one survivor.

It fails silently on writes, which is worse. A symlink is written through to its target, and a hard link shares the inode, so it is written through too. A suite that writes a file — a golden it updates, a fixture it rewrites — therefore writes into the repository being measured. On a fixture whose test rewrites one tracked file, the source came back holding REWRITTEN BY THE SUITE under both links, and untouched under copies.

ditto's own mutant write was always safe, because Overwrite removes a path before writing. That is exactly why this went unnoticed: the only writer anyone checked was ditto.

A refusal now says why the baseline is red, carrying the test command's own output. Without it the message named a red baseline and nothing else — measured at four rounds of elimination over an embedded file that the output named on the first run.

Changed

A release copies the repository into its sandbox. The price is fixed, not proportional: 16.5s over four mutants and 15.0s over nine on a 1960-file tree, because the sandbox is built once per release. That is 85% of the smallest possible run and about 2% of a large one. The earlier "70% slower" was a fixed cost divided by a tiny run.

WithSandboxStrategy reaches "link" and "hardlink" for a caller who has measured that their suite writes nothing.

Measured, and unchanged

The eight exact counters in perf/baseline.json did not move. The golden release output is byte-identical, and so is the one holding the command to the library.

TestASandboxWriteDoesNotReachTheSource is the new guard, and it was watched refusing under both link strategies before it was trusted.

What ran

Tagged on 3d4e8d5, the exact merged head, after a green run on it: CI (1.27) and CI (latest) both pass on ubuntu-latest, with CodeQL, Analyze (go) and OSV-Scanner. SonarCloud reports neutral — inherited from upstream, and not this project's bar.

docs/experiments/the-sandbox-is-a-reference.md carries the hypotheses, including the two this round refuted and the one it withdrew.

v0.4.0 — ditto runs as a command, and knows what a staged change justifies

Ditto took a *testing.T, so anything that wanted to drive a release had to build a test to get in. That is where the wrappers came from. This is the command they were standing in for.

Breaking

• The Go floor is 1.27, up from 1.25. Nothing in the code needs it; the toolchain does.

Added

• ditto staged — mutates exactly what a staged change justifies: which files the index touches, which of their bytes, and it runs the suite against a checkout of the index, not of the working tree.

  Measured: with the release pointed at the worktree instead, and one tracked file left dirty and unstaged, 7 of 8 verdicts moved — 0.13 against 1.00 for the identical mutants of an identical file. Checking that the staged files are clean does not cover it, because the file that moved them was never staged.

  Held against the wrapper it replaces, on two staged changes in a real repository: 4 / 1 / 3 on one file and 9 / 5 / 4 on two, identical on both paths. The second case is the control — the numbers had to move, and both moved together.

• cmd/ditto, with run and staged: go install github.com/Disble/ditto/cmd/ditto@v0.4.0. Policy stays with the caller — --threshold, --test-command, --exclude-prefix, --dry.

• ditto.Run, a release with no test binary around it, plus ditto.RunStaged, ditto.PlanStaged and ditto.Verbose().

Fixed

• A red baseline reaches a command as an error, not a panic. A gate must never make its own refusal indistinguishable from being broken.

Unchanged, and measured to be

The eight exact counters in perf/baseline.json did not move — including sandboxesBuiltPerRelease, whose own note records that it is registered through t.Cleanup in Release, which is precisely what this release restructured. The golden release output is byte-identical, and a new golden holds the command to the same bytes as the library.

What ran

Tagged on 33b5b5e, the exact merged head, after a green run on it: CI (1.27) and CI (latest) both pass, along with CodeQL, Analyze (go) and OSV-Scanner. SonarCloud reports neutral — it is inherited from upstream and applies its own gate, which is not this project's bar.

v0.3.0 — the gated path, and survivors you can find

Full entry with every number: CHANGELOG.md. The notes behind each measurement are in docs/experiments/.

Added

Gated() runs a file's mutants from one compilation instead of one each. Starting the test command costs 750–950 ms per mutant regardless of what the suite does, and that fixed toll is the dominant cost of a run.

It is opt-in, and that is measured rather than assumed. On a real repository the two paths report the same 69 mutants, the same 58 killed and the same 11 survivors at identical addresses, for 51 invocations against 69 — but only 18 of 69 mutants gated. The gating rate is a property of the file, ranging from 26% to 72% across the real files measured, because Expand admits comparisons and integer literals and refuses everything else. A compilation paid for a quarter of the mutants is an option, not a default.

A gated run says how much of it came from one compilation. Zero is spelled none, so the line reads as a fact rather than an unfilled placeholder.

Plus a performance baseline that ratchets in both directions, and a golden test that pins what a run says rather than only what it costs.

Fixed

• Gated() was a no-op under go test -v, which is what CI runs. Measured: none of 7 gated under -v before, 4 of 7 after, against a control of 4 of 7 without it.
• A red baseline reported a perfect score. Measured at 4 killed of 4 against 1 of 4 for the same mutants.
• The compiler was found through PATH, and the test command inherited git's addressing. Both are the same shape: something ambient deciding what a subprocess really is.

Changed — breaking

A survivor is now clickable. Every mutant carries path:line:col and the text it replaced, and survivors are listed by address before any diff. 129 of 135 mutants used to be indistinguishable from another by what ditto printed.

The label changed shape, from path → Virus to path:line:col → Virus. Subtest names change with it — anything matching on those strings needs updating; anything reading the exit code does not.

Checks

make lint test.failfast green on every commit — 337 tests, 0 lint issues. CI green on a97e04a, the tagged commit. SonarCloud quality gate passing.

This repository is a fork, so GitHub does not fire Actions on push or pull request; the CI run above was dispatched by hand, which is the only route available. mutation.yml has no dispatch trigger and could not be run.