Checks That Check Nothing

A check that passes having checked nothing

Section 10 of the method. Make every check fail when it is handed nothing to check.

Every recurring failure I have had this year is one shape. Not a check that fails when it should not. A check that reports success without having checked anything, because silence and success look identical and nobody investigates a pass.

I had two in one afternoon, on a release that was otherwise ready.

The preflight script passed having run no tests. It takes the test classes to run as arguments. Given none, it did the restore, the build and every file scan, then printed all checks passed. I read that line, believed it, and pushed. CI caught the real problem eighteen minutes later.

What makes this worth writing down is that the same script already refused the identical hole one level down: a named class matching zero tests was treated as a failure, with a comment explaining that a filter matching nothing still "finishes cleanly, exit 0". The reasoning was right and was applied one level too shallow. The fix is six lines.

if [ ${#classes[@]} -eq 0 ] && [ "$no_tests" -eq 0 ]; then
  echo "no -class given, so no test would run and this would pass having tested nothing."
  exit 1
fi

Nothing compared three files that pin the same version. A release moves a version in one file and the others are forgotten. An integration test caught it, correctly, eighteen minutes into CI. The test was doing its job. Nothing was doing the job of saying so before the build. check-pinned-values.sh in this repository is the general form: give it file and capture pairs and it fails with a diff.

Note what I did not do. The test that caught it pins the version as a literal on purpose, so that bumping the constant is a conscious act rather than something carried along automatically. Deleting it to stop the noise would have removed the only thing that noticed. The new check runs earlier; it does not replace it.

The general form. For every check you own, ask what it does when handed nothing to do. Not what it does when it finds a problem, which is the case you designed and tested. A test runner given an empty filter. A linter given no files. A scanner pointed at a directory that moved. A comparison where both sides are empty strings, which are equal. Each of those exits 0, and an exit code of 0 is the only thing anyone reads.

Then make it fail before you trust it. Revert the fix and watch the gate go red, or feed the check the exact mistake it exists to catch. Both of the gates above were written that way, and testing the empty-capture case is what found that two unreadable files would have compared equal and passed. That is the bug this whole section is about, sitting inside its own fix.

The worst one I have had was a check that did not exist. Section 2 above has described a mutation check since early September: name the production line a test depends on, revert it, the test must fail. Question 1 of the critic's list asked whether that had happened. For two weeks critics answered yes.

There was no script. The preflight file had no revert step and no hunk parsing, and not one of the last forty merged changes carried a binding. The check I wrote to catch gates that cannot fail was one, and it had the strongest possible output: a reviewer affirming it in writing, on every change.

Two things in that are worth more than the fix.

An item phrased as a fact can be answered from the description. "Was the mutation reverted" is a claim about the world that a reader satisfies by believing the change description. "Did the holdout check run, and is its output here" names an artefact that either exists or does not. Phrase every checklist item as the second kind. If an item can be satisfied by an opinion, it will be.

Nothing found it. A person did, reading the script against the document that described it. When the answer to "what caught this" is a name rather than a mechanism, the mechanism is the finding.

The replacement holds a hunk out in a throwaway copy, rebuilds, and requires the named test to fail, then restores and requires it to pass. One decision in it matters more than the rest: a held-out tree that does not compile is inconclusive, not a pass. Reverting one hunk of a typed language usually breaks the build, and a tree that does not build runs no tests. Score that as success and the gate is green forever over a check that never ran, which is this section's bug reappearing inside the fix for this section's bug. It also fails on an unclaimed hunk, so a change that tests nothing has to say so in writing next to the diff rather than simply staying quiet.

It ships with a fixture suite that runs a known one-hunk change past eleven cases, every failure path included, and preflight fails if any of them stops returning its exit code.

What running it has actually shown, after nine runs on one repository. Cost was 24 to 62 seconds per binding, two builds and two test runs included, integration tests on containers at the top of that range. I had expected minutes and had written slowness down as the thing most likely to kill it. It is not.

Of the six runs against real merged changes, three passed and three were inconclusive. A pass is worth reading precisely: the named tests failed while their hunk was held out and passed again once it was restored, so the test does depend on the code it claims to. Three of three inconclusives were the same cause, a hunk that something else referred to, so reverting it did not compile and no test ran.

It caught one false binding, on a change I built to be catchable: a line that could never execute, with a test bound to it that passed either way. One catch in nine runs, and the eight others were either honest passes or inconclusive. That is not evidence it finds defects in the wild. It is evidence the mechanism does what it says when the question can be asked.

Running it found four defects in itself and one thing it had never supported. It resolved the repository from its own file path, so a copy run from anywhere else reported a git error for what was a path bug. It spent two builds discovering that a whole-file hunk cannot be held out, which the diff header says for free. It reported a stopped Docker as a broken branch. It said "all bindings held out and failed as required" on a change where every hunk was declared untested and nothing had been held out, which is this whole document's bug appearing inside its own fix. And it never parsed the none: declaration, although the change that introduced it put none: in its own description. Five findings, none of them from review, all of them from use.

Nine runs is not twenty, and only six were against changes I did not write for the purpose. The number that would make me delete it still stands: twenty with no catch, or more than half coming back inconclusive.

---

Previous: Half of what an agent makes is not text | Home | Next: The machine has a ceiling too