Holdout Check

Holdout Check

holdout.sh asserts that a test added by a change notices the change it claims to test. It holds the production hunk out, rebuilds, and requires the named test to fail. Then it restores the hunk and requires the test to pass.

bash holdout.sh --spec change-description.md
bash holdout.sh --spec change-description.md --dry-run

The spec

A fenced holdout block in the change description binds each new test to the hunk it depends on.

```holdout
- test: Some.Tests.RedeemTests
  breaks: src/Features/Redeem.cs "RevokedAt is not null"
- untested: src/Features/Endpoints.cs #1
  why: route registration, covered by the inventory check
```

• test: and breaks: name a test class and the hunk it depends on. The anchor is line content, not a line number, so a binding survives a rebase.
• untested: and why: declare a production hunk with no test, in writing.
• Anything else in the block is an error, not a skip.
• A production hunk that nothing claims fails the check.

Exit codes

Code	Name	Meaning
0	pass	Held out, the test failed, restored, the test passed again
1	caught	A test passed with its hunk held out
2	unresolved	The anchor matched no hunk or several, or zero tests ran
3	inconclusive	The held-out tree did not build, so nothing was proven

A held-out tree that does not compile is inconclusive, never a pass. A tree that does not build runs no tests, and scoring that as success would leave the gate green over a check that never ran.

Adapting it

Two settings name your build and your single-test run. The defaults are for .NET.

Setting	Default
HOLDOUT_BUILD_CMD	dotnet build --configuration Release --no-restore
HOLDOUT_TEST_CMD	dotnet test --no-build --configuration Release --filter FullyQualifiedName~
HOLDOUT_BASE	origin/master

The test command is given one argument, the test id from the spec, and must run that and nothing else.

It runs in a throwaway worktree and never changes the caller's tree.

When it can answer

It answers when the hunk can be removed without breaking a reference to it. That means a change to code that already existed, which is most bug fixes and hardening work. It does not mean a file the change adds outright, or a new method other code already calls.

Size is not the predictor. A 197 line hunk held out cleanly because what it replaced was self-contained. A 41 line new method failed because the file called it three lines later.

What running it has shown

From nine runs on one repository (README section 10):

• 24 to 62 seconds per binding, two builds and two test runs included.
• Of six runs against real merged changes, three passed and three were inconclusive. Every inconclusive was a hunk something else referred to.
• It caught one false binding, on a change built to be catchable.

That is evidence the mechanism does what it says when the question can be asked. It is not evidence that it finds defects in the wild. It does not replace the critic: a check cannot see what a test does not test.