test: enumerate the fixture corpus instead of growing it by example by davydog187 · Pull Request #67 · elixir-ai-tools/just_bash

davydog187 · 2026-08-05T17:33:51Z

Groundwork for a differential testing story that finds bugs before a production agent does, plus the first enumerated matrix and the bugs it found.

Why

The recent fix wave (#56, #57, #60, #63) was found reactively: an agent driving the sandbox returned a wrong answer, a human noticed, an issue got filed. There is no proactive enumeration. The obvious remedy is "run real bash in a container and compare" — but that already exists and works. mix bash_fixtures records real bash in Docker and fixture_test.exs byte-compares stdout/stderr/exit across 33 suites.

The gap is what the corpus contains. One measurement settles it:

test/fixtures/bash_cases/date.json has 30 hand-written cases and covers none of %F, %D, %T, %R, -I or -v — the exact directives #63 had to fix.

The existing, unmodified oracle would have caught that bug with ~25 lines of JSON. It shipped because nobody wrote down strftime's alphabet. The corpus grew by example — one case per bug already found — so a directive nobody had thought about had no case, and an absent case is indistinguishable from a passing one.

A command's conversion and flag sets are finite. They are not a space to sample with a fuzzer; they are a list to enumerate.

What's here

1. The corpus verifies its own content-addressing (2bffbbd)

Cases and recordings are joined by content_hash, and nothing in the repo computed it — it was only ever read, as an opaque join key. Two consequences: a case couldn't be added programmatically (the only way in was to hand-compute a 16-hex value by an unknown recipe), and a stored hash that is never re-derived can't detect its own staleness.

62 of 968 hashes had drifted. Re-recording them produced no behavioral change — the expectations were right for the current scripts, so the hazard was latent rather than realised. It no longer is: JustBash.Fixtures owns the digest, fixture_test.exs recomputes rather than trusts it, and a stale hash or collision now fails compilation instead of silently mispairing.

mix bash_fixtures.verify — the same checks offline. Because the hash comes from the inputs, "edited a script but never re-recorded" is detectable with no Docker at all, which is what lets CI stay hermetic.
mix bash_fixtures.rehash — adopts edited scripts. Deliberately leaves expectations unclaimed so verify says so and the re-record diff shows what the cases should have been asserting.
The recorder piped Docker's stderr into Jason.decode!. Any byte Docker writes there — image platform mismatches, pull progress — corrupted the payload silently, since a truncated parse fails far from its cause. It now writes to a mounted file.
Only recording inputs are hashed. Renaming a case, or widening what it tolerates, must not orphan a recording that is still byte-for-byte correct — the property fixture_test.exs:27 already claimed in a comment but nothing enforced.
Drops the unused locked:true mechanism: a recording that silently contradicts real bash is the artifact that hides drift.

2. mix bash_fixtures.gen + the date matrix (e3336f6, e50c105)

182 enumerated cases against real GNU date. In a command that already had 30 hand-written cases, they found 40 divergences:

class	count	examples
conversion printed verbatim at exit 0	21	`%c %g %G %k %l %q %r %U %V %W %x %X %z`, every field-flag form (`%-d %_d %0d %^a %:z %::z %:::z`)
flag silently discarded, returns today at exit 0	7	`-v+6m`, `-v-1d`, `-Z`, `--not-a-flag`, `-j`, `-jf`
wrong format	1	`-R` emitted the default format, not RFC 5322
`-d` input rejected	2	`@SECONDS`, offset-less ISO
flag rejected that GNU implements	1	`-r FILE`

The first class is the same bug #63 fixed for %F. That fix closed instances, not the class — 21 more conversions were still printing themselves at exit 0. The second is #64 exactly, and it is the failure that had an agent file a goal six months in the past and report success.

All fixed. date -r now reports a file's mtime from the VFS, with errors through FS.strerror/1 so descending through a regular file reports ENOTDIR rather than the hardcoded "No such file or directory" that 33 command modules still use for every error kind.

Two things the oracle corrected mid-fix

Both worth keeping, because both are cases where reading the docs would have left the bug in:

The existing composition property in property_test.exs caught the first attempt treating %#%! as a modified %%. Real date refuses % as a conversion for a flag run: it emits the flags literally and starts a fresh directive at the %, so %-%d is %-15. The property was right.
Folding flags one at a time renders %-0d as 5. Real date renders 05 — the last padding flag wins outright, so padding is now decided once from the whole run.

Enumeration lessons, encoded

One base instant hides half the bugs. At day 15 and hour 13, %-d, %_d and %0d all render "15" and every padding bug hides. Every conversion is recorded at two instants, one with a single-digit day and a morning hour — which is what surfaced %_3d.

Gaps are recorded, never omitted. 13 divergences remain, each with the reason it stands: GNU's -d relative grammar ("6 months ago", "next tuesday") is unimplemented, -j/-f are documented BSD spellings GNU rejects, and %N pads with trailing spaces under _ and - unlike every other conversion. A known_gap opt marks them; it rides in opts, which the digest excludes, so marking or closing a gap never invalidates a recording. A recorded gap is countable and has a reason attached; an absent case looks like a passing one, which is how this class survived #63.

Verification

All five gates from CLAUDE.md: mix format --check-formatted, mix compile --warnings-as-errors, mix credo --strict (only the intentional test-fixture finding), mix dialyzer, mix test — 4219 tests, 0 failures, 14 skipped. mix bash_fixtures.verify: 34 suites sound. Corpus is now 1150 recorded differential cases.

Closes #64.

Follow-up work is tracked in #70 (the remaining matrices, oracle hardening, oracle-free invariants, the Oils subset and CI topology), with the two prerequisites split out as #68 (unknown flags absorbed as operands) and #69 (builtin raises escape exec/2; Limit has no wall-clock bound).

Cases in test/fixtures/bash_cases are joined to the bash output recorded in test/fixtures/bash_expected by content_hash, a digest of the case inputs. Nothing in the repo computed that digest. It was only ever read, as an opaque join key, which left two holes: - A case could not be added programmatically. fixture_test.exs raised a CompileError on a missing hash and pointed at a script that never emitted one, so the only way in was to hand-compute a 16-hex-char value by an unknown recipe. - A stored hash that is never re-derived cannot detect its own staleness. Edit a script, leave the hash, and the case keeps passing while asserting against output recorded for a different script. 62 of 968 cases had drifted that way, across cp, head_tail, mv, real_world (29/29), special_variables and wc. Re-recording them shows no behavioral change: the expectations were right for the current scripts, so the hazard was latent rather than realised. It no longer is. JustBash.Fixtures is now the one definition of the digest, recovered from the corpus so the 906 sound hashes keep validating: sha256({"files":<compact json>,"script":<compact json>}) |> hex |> first 16 Only the fields that feed a recording are hashed. Renaming a case, or widening what it tolerates via opts, must not orphan a recording that is still byte-for-byte correct — the property fixture_test.exs already claimed in a comment but nothing enforced. fixture_test.exs now recomputes the hash instead of trusting it, and refuses to compile on a stale hash or a collision. Both make every assertion in a suite untrustworthy, so they fail the build rather than one test. Two companion tasks, and one recorder fix: - mix bash_fixtures.verify — the same checks as a report, offline. Because the hash comes from the inputs, "edited a script but never re-recorded" is detectable with no Docker at all, which is what lets CI stay hermetic. - mix bash_fixtures.rehash — adopts edited scripts. Deliberately does not touch expectations: it leaves them unclaimed so verify says so and the re-record diff shows what the cases should have been asserting. Rewrites through ordered JSON so a hash change is a one-line diff, not a reordered file. - mix bash_fixtures piped Docker's stderr into Jason.decode!. Any byte Docker writes there — image platform mismatches, pull progress — corrupted the payload, silently, since a truncated parse fails far from its cause. The runner now writes to a mounted file. It also checks at record time that every live case is covered, against the container it just ran. Drops the locked:true mechanism. No fixture used it, and a recording that silently contradicts real bash is the artifact that hides drift. 4033 tests, 0 failures. format, compile --warnings-as-errors, credo --strict and dialyzer clean.

…n flags `date +%F` printing the literal "%F" at exit 0 was fixed in 274e86b by adding %F, %T, %R and %D. Enumerating the rest of the alphabet against real GNU date shows the fix closed instances, not the class: 21 more conversions were still printing themselves at exit 0, and 7 flags were still being silently discarded. The reason nobody noticed is visible in the corpus. date.json has 30 hand-written cases and covers none of %F, %D, %T, %R, -I or -v — it grew by example, one case per bug already found, so a directive nobody had thought about had no case and looked exactly like a passing one. A command's conversion and flag sets are finite. They are not a space to sample; they are a list to enumerate. mix bash_fixtures.gen writes that enumeration out as ordinary case files, so recording and comparison work as they already do. `mix bash_fixtures.gen date` produces 179 cases against real GNU date, which found: - 21 conversions emitted verbatim at exit 0: %c %g %G %k %l %q %r %U %V %W %x %X %z, and every field-flag form (%-d %_d %0d %^a %#a %:z %::z %:::z). - 7 flags silently discarded, each returning today's date at exit 0: -v+6m, -v +6m, -v-1d, -Z, --not-a-flag, and -j/-jf. This is issue #64 exactly, and it is the failure that had an agent file a goal six months in the past and report success. Unknown flags are now refused with GNU's own message. - -R emitted the default format rather than RFC 5322. - -d rejected two forms it should accept: @seconds, and an ISO timestamp with no offset (DateTime.from_iso8601/1 requires one). Each base instant only exposes what it happens to distinguish, which is its own version of the same blind spot: at day 15 and hour 13, %-d, %_d and %0d all render "15" and every padding bug hides. Every conversion is therefore recorded at two instants, one with a single-digit day and a morning hour. Two things the oracle corrected mid-fix, both worth keeping: - The existing composition property in property_test.exs caught the first attempt treating %#%! as a modified %%. Real date refuses % as a conversion for a flag run: it emits the flags literally and starts a fresh directive at the %, so %-%d is "%-15". The property was right; those sequences are now pinned in the matrix rather than resting on one generated example. - Folding flags one at a time renders %-0d as "5". Real date renders "05" — the last padding flag wins outright, so padding is decided once from the whole run instead. 13 divergences remain, each recorded with the reason it stands rather than left out of the corpus: GNU's -d relative grammar ("6 months ago", "next tuesday") is unimplemented; -j and -f are documented BSD spellings GNU rejects; and %N pads with trailing spaces under _ and -, unlike every other conversion. A recorded gap is countable and has a reason attached. An absent case looks like a passing one, which is how this class survived the last fix. known_gap rides in opts, which the digest excludes, so marking or closing a gap never invalidates a recording. 4212 tests, 0 failures, 14 skipped. All five gates clean.

Refusing unknown flags in the previous commit refused `-r` along with them, but `-r FILE` / `--reference=FILE` is a real GNU flag: it reports the file's modification time. Rejecting it traded a silent wrong answer for a wrong error, which is better but still wrong, and it left issue #64's third reproduction case failing for a new reason. The VFS records mtimes, so `-r` reads from the sandbox rather than the host clock. Errors go through FS.strerror/1, so descending through a regular file reports ENOTDIR rather than the "No such file or directory" that 33 command modules still hardcode for every error kind. Also distinguishes "flag needs an argument" from "no such flag" for -d, -f and -r. The reason to refuse a flag at all is that the message tells the caller what to do differently, which a wrong message does not. -r's success path cannot go in the date matrix: the mtime of a file created during recording belongs to the recording clock, so the two engines cannot agree on it by construction. Its error paths are enumerated there; the success path is a unit test against a seeded mtime. Closes #64.

The date matrix enumerated two alphabets — every conversion, and a handful of flag spellings — and never crossed them. That is how `%-T` shipped rendering "9:05:03": no case ever asked a flag and a conversion in the same breath, and an absent case looks exactly like a passing one. Crossing them takes the matrix from 182 to 672 cases and finds 114 divergences, all of them exit 0 with a wrong answer: * a padding flag mangled the compound conversions — `%-T`, `%-D`, `%-R`, `%-r`, `%-x`, `%-X`, `%_D` — by stripping characters from a rendered string rather than restating a numeric field's width * `%E` and `%O` printed themselves verbatim, though GNU accepts them for 51 of the conversions crossed here * `#` was treated as a synonym for `^`, so `%#p` was "AM" and `%#Z` "UTC" * `%z` was a literal rather than a signed number, so `%-z` kept the padding it was asked to drop and `%8:z` ignored its width entirely * a width was a floor rather than a replacement, so `%1d` was "05" The renderer now models a conversion as a field — a number with a value, digit count and pad character; a signed zone offset; text; or a sub-format — and lets the flags restate that field rather than edit its output. The rules it follows are the ones the recording implies, not the ones the documentation suggests: a width replaces the conversion's own digit count instead of raising it, the last padding flag wins outright, a modifier drops the padding flag, and a compound forwards its flag to year fields only, which is why `%-D` is "01/05/5" while `%-x` is "01/05/05". The E and O acceptance sets are transcribed from real date rather than derived: they overlap without containing each other, and `%Eq` renders where `%Oq` does not. Three base instants where there were two, plus a fourth for the year fields: June cannot distinguish `%-j` from `%j`, and no modern year can show `%_Y` padding at all. Argument parsing gets the other half. Refusing unknown flags had also refused `--` and the separate-argument spellings `--date VALUE` and `--reference VALUE`, which GNU accepts. `-d` with `-r` silently answered one of the two questions asked. And `-d @N` checked the shape of the number but not whether it was a representable instant, so a large enough one raised out of the command instead of being refused.

…ails Three ways the corpus could report success it had not earned. A `known_gap` was a skip. A skip can only ever report the divergence someone wrote down; it cannot notice the divergence is gone, so a gap closed by an unrelated fix stays marked forever and the count stops counting anything. The assertion is inverted instead — a gap case runs and fails when its output *matches* the recording. It found a stale marker on its first run: `-jf` is one clustered token that both engines refuse on the same byte, never the BSD `-j -f` pair it was named for. That pair is now enumerated separately, and it is the real gap. `verify_recorded!` did not raise despite the name, and the recording it had just rejected was written anyway, over a good one, at exit 0. Nothing downstream could tell that recording from a clean one: the file it would have replaced is still there and still passing. It now returns an error, the unusable recording is not written, and the task raises at the end naming every suite that failed. A case with no `"script"` raised a FunctionClauseError from inside `validate/2`, which reads as a bug in the checker rather than a malformed case file. It says which now.

ivarvong approved these changes Aug 5, 2026

View reviewed changes

davydog187 added 5 commits August 5, 2026 17:27

davydog187 force-pushed the test/fixture-corpus-integrity branch from 0e4f9d5 to 68809e5 Compare August 5, 2026 22:19

davydog187 merged commit 82ee9e5 into main Aug 5, 2026
4 checks passed

davydog187 mentioned this pull request Aug 6, 2026

fix: exec/2 returns a shell result instead of raising, and always terminates #73

Merged

Navigation Menu

Uh oh!

test: enumerate the fixture corpus instead of growing it by example - #67

test: enumerate the fixture corpus instead of growing it by example#67
davydog187 merged 5 commits into
mainfrom
test/fixture-corpus-integrity

davydog187 commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

davydog187 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What's here

Two things the oracle corrected mid-fix

Enumeration lessons, encoded

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davydog187 commented Aug 5, 2026 •

edited

Loading