Skip to content

v0.5.0 — a sandbox is a copy, not a reference

Choose a tag to compare

@Disble Disble released this 27 Aug 23:13

Two entries, one defect seen from two sides. Both were found by measuring rather than by reading, and the second one corrects the first fix that shipped in this branch.

Fixed

A sandbox mirrored the repository as symlinks. That is a reference to the tree, not a copy of it, and the two part company in two ways that matter.

It fails loudly on reads. Go refuses to embed an irregular file, so no package with a go:embed directive could build in a sandbox at all — every release before this one silently could not measure them. Found on a real repository where it reported 9 mutants, 9 killed, a perfect 1.00 for a package that never compiled, because a failing command is how ditto recognises a killed mutant. The truth is 8 killed and one survivor.

It fails silently on writes, which is worse. A symlink is written through to its target, and a hard link shares the inode, so it is written through too. A suite that writes a file — a golden it updates, a fixture it rewrites — therefore writes into the repository being measured. On a fixture whose test rewrites one tracked file, the source came back holding REWRITTEN BY THE SUITE under both links, and untouched under copies.

ditto's own mutant write was always safe, because Overwrite removes a path before writing. That is exactly why this went unnoticed: the only writer anyone checked was ditto.

A refusal now says why the baseline is red, carrying the test command's own output. Without it the message named a red baseline and nothing else — measured at four rounds of elimination over an embedded file that the output named on the first run.

Changed

A release copies the repository into its sandbox. The price is fixed, not proportional: 16.5s over four mutants and 15.0s over nine on a 1960-file tree, because the sandbox is built once per release. That is 85% of the smallest possible run and about 2% of a large one. The earlier "70% slower" was a fixed cost divided by a tiny run.

WithSandboxStrategy reaches "link" and "hardlink" for a caller who has measured that their suite writes nothing.

Measured, and unchanged

The eight exact counters in perf/baseline.json did not move. The golden release output is byte-identical, and so is the one holding the command to the library.

TestASandboxWriteDoesNotReachTheSource is the new guard, and it was watched refusing under both link strategies before it was trusted.

What ran

Tagged on 3d4e8d5, the exact merged head, after a green run on it: CI (1.27) and CI (latest) both pass on ubuntu-latest, with CodeQL, Analyze (go) and OSV-Scanner. SonarCloud reports neutral — inherited from upstream, and not this project's bar.

docs/experiments/the-sandbox-is-a-reference.md carries the hypotheses, including the two this round refuted and the one it withdrew.