v0.17.0

Agent-first surface, first slice — and detecting suites that cannot fail.

⚠️ Breaking, and the warning you saw was wrong

Three deprecations are removed. They were deprecated in v0.13.0 with a warning that read Removal in v0.14.0. — and then shipped in five further releases still saying that.

If you read that warning carefully, you reasonably concluded the removal had already happened and your spec was fine. "We warned you" is not an honest defence here; the warning named a version two behind the one you were running.

Removed	Use instead
faultbox generate	faultbox plan --suggest
stdout() / stderr()	observe.stdout / observe.stderr
json_decoder() / logfmt_decoder() / regex_decoder()	decoder("json" | "logfmt" | "regex")

The names still resolve. Using one now fails at spec load with an error naming its replacement and the release that removed it — not a bare undefined: stdout, which is true and useless.

stdout() was removed in v0.17.0 — use observe.stdout instead.
It was deprecated in v0.13.0 (RFC-044) and warned on every run since

The bug that motivated this release

A CI spec exercised a broken Postgres client on every pull request for three releases, and passed.

env = {"POSTGRES_HOST_AUTH_METHOD": "trust", ...}   # removes the credential path

resp = pg.main.query(sql = "SELECT 1")
assert_true(not resp.ok, "expected failed query under injected fault")

It asserts the query fails. A client that cannot connect at all satisfies that identically to the injected fault. Its own comment stated the intent — "so this test doesn't depend on authentication round-tripping."

A careful test. The care is what hid the bug. There was no positive control: nothing anywhere asserted that a Postgres step succeeds.

Added — diagnostics for suites that cannot fail

NO_POSITIVE_CONTROL — an interface is stepped, but no test ever asserts a step on it succeeds.

Suite-level, which is what makes it new. A single fault-injection test asserting failure is correct and normal; a suite where that is the only assertion an interface ever receives proves nothing. No per-test lint can see that.

TEST_NO_ASSERTIONS — a test passed having evaluated nothing.

Both are warnings. They found two vacuous specs in this repository — one in the CI golden corpus — each carrying the same false belief in a comment: "the assertion is the absence of a panic." A failed step returns ok = False; it does not raise.

Added — faultbox check

faultbox check spec.star --format json

Validates without running: no processes, no image pulls, no Docker. Milliseconds against the tens of seconds a run costs. The runtime could always do this — it simply was not exposed, so the only way to learn a spec was malformed was to run it and wait for containers.

Findings carry machine-readable codes and a suggested next move. Also the MCP tool check_spec, running the identical code path.

It will not tell you whether your suite is meaningful — that needs a run. check finds specs that are wrong; faultbox test finds specs that are empty.

Added — error-code taxonomy

Eight codes over spec-load and infrastructure failures, each carrying the reader's next move. Full reference: diagnostic-codes.md.

Implemented as typed errors, not by matching message text — that shortcut would reproduce the exact fragility this release removes, one layer down. So adoption is incremental and Classify reports an uncoded error as uncoded rather than guessing. A gap in the taxonomy is discoverable; a wrong code is something an agent acts on.

Fixed — a diagnostic nobody could see was wrong

Per-test diagnostics have existed since v0.12 but were only ever written to JSON. Nobody running faultbox test interactively has seen one in four releases. They now print.

Making them visible immediately exposed that FAULT_FIRED_BUT_SUCCESS was miscalibrated. Its heuristic — a fault fired and the test passed — describes the single most common correct shape in the tool:

def test_api_cannot_reach_db():
    def scenario():
        resp = api.post(path="/data/failkey", body="value1")
        assert_eq(resp.status, 500)              # degradation, asserted
    fault(api, connect=deny("ECONNREFUSED"), run=scenario)

That test is right, and the diagnostic called it suspicious — for five releases, unnoticed, because nobody ever saw it. Now requires the test to have asserted nothing.

A diagnostic nobody sees is a diagnostic nobody calibrates. Five other codes shipped in that same silence and are now visible without the individual scrutiny this one received. They produced no false positives across the corpus, but that is weaker evidence and is recorded as such.

Also

• MCP contract tests. faultbox mcp had no test coverage at all, on the one surface whose declared primary user is an agent.
• TestResult.assertions in JSON — a green result means something different depending on whether anything was checked.
• Driving Faultbox as an agent — the loop, the JSON shapes, the trap that cost three releases.

What was cut, and why that matters

A third diagnostic, STEP_RESULT_DISCARDED, was fully implemented and then cut. Across 54 specs it produced 15 findings of which 13 were legitimate — steps used for their side effect, where the result genuinely does not matter:

# GET /trigger?leak=clock — leaker performs a raw clock_gettime before responding.
leaker.main.get(path = "/trigger?leak=clock")

The rule that cut it was written down before the measurement, precisely so the result could not be rationalised afterwards. Both true positives were already caught by the other two diagnostics.

A diagnostic that fires on correct code gets muted, and a muted diagnostic is worse than an absent one.

Verification

go build + go vet + go test -race ./... green; cross-compile clean on four targets. NO_POSITIVE_CONTROL fires on a reconstructed known-bad spec and stays silent on all eight poc/protocol-audit specs; both new diagnostics show zero false positives across 11 runnable specs.

Full detail in CHANGELOG.md and RFC-052.