v0.16.1

The protocol step-client audit — eleven fixes, ten of them pre-existing and invisible to the test suite.

v0.16.0 fixed two Postgres bugs that survived because no spec had ever asserted on the result of a database step. This release repeats that audit across the other twelve protocols: one spec per protocol, real server, every result checked (poc/protocol-audit/).

Several of the bugs it found were hiding behind each other.

Upgrade if you use MySQL, MongoDB, Redis with a password, or NATS

You would have seen	Cause
invalid connection on every MySQL step	No password, no database in the DSN
MySQL query() hanging forever, run never finishing	Proxy over-read the result set
{"id": ""} where you stored {"id": 1}	Dict values coerced to strings
nats: expected 'PONG', got 'PONG'	Proxy stripped NATS's CRLF framing
NOAUTH from Redis	No AUTH sent
ready() timing out against a healthy service	URL handed to plugins that dialled it verbatim
no space left on device, blamed on the wrong spec	Every container leaked its data volume

The bug behind a bug

Fixing MySQL's credentials let a step reach the command phase for the first time — and immediately exposed a proxy that hung forever on every result set. forwardResponse peeked to find the response end, the peek consumed the terminator, and the next iteration issued a deadline-free read for a packet that would never arrive. exec() was fine; every query() wedged.

The hang landed in teardown, after the test body had finished, so no per-test timeout could fire. All twelve proxies had the same unbounded Stop(), so one stuck handler hung a whole run. Now bounded, and it emits proxy_stop_timeout rather than failing quietly.

Self-consistently wrong

Dict values were read with a coercion returning "" for anything non-string, so document = {"id": 1} stored {"id": ""}. It hid because a filter of {"id": 1} was mangled identically, matched the mangled document, and the round trip looked correct.

List arguments arrived as Starlark source text — so mongodb.insert_many had never run, and Redis's documented command(args=[...]) dropped every argument.

Worth knowing even if you skip this release

Faultbox removed its containers but not the anonymous volumes every stock database image declares. Measured on one dev VM after a session of runs: 290 orphaned volumes, 18.7 GB, a full 30 GB disk, and every later container failing with no space left on device.

The failures name whichever spec runs next, so the shape is a flaky test that is really a resource exhausted three specs earlier. If you run Faultbox in CI against database images, docker volume prune -f is worth checking regardless of when you upgrade.

The lesson, now enforced in docs

A failing protocol step returns ok = False — it does not raise, because a failing dependency is frequently what a fault-injection spec is deliberately provoking. The cost is that an ignored result is indistinguishable from a successful one.

r = db.main.exec(sql = "INSERT INTO orders VALUES (1)")
assert_true(r.ok, "insert failed: %s" % r.error)

That is now Pattern 0 in the patterns guide, alongside the subtler form: a watch() audit that passes on a service's boot I/O while its workload fails silently. Completeness of observation and occurrence of the workload are different claims.

Also

• Credentials from env= for MySQL, Redis (AUTH), MongoDB, ClickHouse, Cassandra and NATS — declared once, used by steps and healthchecks alike
• args= now overrides a container's CMD; it was accepted then silently dropped
• NATS publish confirms delivery instead of discarding the flush error
• ready() retries for MongoDB and Cassandra, which previously made one attempt at the instant the port bound — Cassandra could never have passed
• Docs: setup-trace was missing from the CLI reference entirely; the feature manifest had no rows for v0.14.1 or v0.16.0; mysql.md documented a topology its own client could not connect to. 36 broken links fixed.

Not claimed

http2, udp and grpc still have unit coverage only — no real-server spec. Recorded as a gap in the feature manifest, because reaching four protocols with a real server found four bugs no unit test saw. "Shares the same code path" is not evidence that a path works.

The netfault flake is hardened, not proven: after the fix it failed once more under full-suite load, then 28 runs clean.

Verification: go build + go vet + six consecutive clean full go test -race ./... suites; cross-compile clean on four targets. End-to-end in Lima on kernel 6.8 — postgres 2/2, mysql 2/2, redis 3/3, redis-auth 1/1, mongodb 5/5, clickhouse 2/2, nats 2/2, cassandra 2/2, plus poc/example, poc/demo-container and poc/kafka-rfc014. NATS ran 18 consecutive clean times to confirm its intermittent failure was gone. The volume-leak fix was verified by the corpus itself: eleven container-heavy specs back to back left the VM at 0 volumes and unchanged disk use.

Full detail in CHANGELOG.md.