Skip to content

name the case, the reason, and the difference in test failures - #643

Merged
kacy merged 1 commit into
mainfrom
testing-assertions
Aug 3, 2026
Merged

name the case, the reason, and the difference in test failures#643
kacy merged 1 commit into
mainfrom
testing-assertions

Conversation

@kacy

@kacy kacy commented Aug 3, 2026

Copy link
Copy Markdown
Owner

three defects in std.testing made a table of cases unpleasant enough to be worth avoiding — which is the actual reason exhaustive negative coverage doesn't get written, more than any missing helper.

a failure didn't say which case failed. check() took a name but compared only ints; check_str only strings. anything else fell back to assert_eq, whose report is literally named "assert_eq" — over a loop of seven vectors it named neither the row nor the parameters, leaving two hex strings to diff by eye. check is now generic and carries the label, and check_str routes through it, so the two hand-specialized copies became one implementation.

a text mismatch reported the two lengths. got 412 chars, want 409 tells you nothing about a difference four hundred characters in. it now reports the first differing offset with a window of each side:

first differ at char 3: got "def", want "Xef"

a rejection couldn't say why. every negative test in the crypto modules is assert(r.is_err), which passes just as happily when the call fails for a reason nobody intended — a malformed fixture, a renamed field, a check that ran before the one under test. assert_err_contains(message, part) takes the error's message and the text it should mention. that one is a correctness gap rather than ergonomics: several existing negative tests would pass today if the failure moved to an unrelated cause.

the boundary this uncovered, now documented

while adopting the labelled check inside a colocated test block, the block's count silently dropped — std.testing keeps its own tally, so calling these helpers inside a test block records into a count that block never prints, and the assertion quietly stops counting. i caught it because a suite went from 10 tests to 8.

the module header now states it: built-in assertions inside test blocks, these in a test script that ends with done().

that also bounds what this PR buys. the labelled form helps tests/cases-style scripts today; colocated module tests still can't share a table-driven runner, because assert_eq is undefined outside a test block. that remains compiler work and is the next thing worth doing — but it is genuinely separate, and worth knowing that inline tables inside a test block work fine at one-module scale (the argon2 and jwt vector tables in #642 are written that way).

what was tested

  • tests/cases/test_std_testing_helpers.pith exercises each new assertion, including the labelled form reporting by case name and the text diff; expected output refreshed
  • make run-regressions-only: 304 passed, 0 failed
  • make run-examples: 105 passed, 0 failed — examples/stdlib_test.pith output byte-identical, confirming the generic check is compatible with the 311 existing int call sites
  • tests/cases/test_suite.pith (the heaviest check consumer): 46 passed
  • doc coverage clean

notes

  • check_* names are kept rather than deleted. removing them would mean rewriting 400+ call sites across four files for no user-visible gain; unifying the implementation behind them gets the dedup without the churn.

three defects made a table of cases unpleasant enough to be worth avoiding,
which is the real reason exhaustive negative coverage does not get written.

a failure did not say which case failed. check() took a name but only
compared ints, and check_str only strings, so anything else fell back to
assert_eq, whose report is literally named "assert_eq" — over a loop of
seven vectors it named neither the row nor the parameters. check is now
generic and takes the label, and check_str routes through it, so the two
hand-specialized copies are one implementation.

a text mismatch reported the two lengths. "got 412 chars, want 409" tells
you nothing about a difference four hundred characters in; it now reports
the first differing offset with a window of each side.

and a rejection could not say why it was rejected. every negative test in
the crypto modules is `assert(r.is_err)`, which passes just as happily when
the call fails for a reason nobody intended — a malformed fixture, a
renamed field, a check that ran before the one under test.
assert_err_contains takes the message and the text it should mention.

the module header now states the boundary that cost me a silent failure
while writing this: these helpers keep their own tally, so calling them
inside a colocated `test` block records into a count that block never
prints and the assertion quietly stops counting. built-in assertions inside
test blocks, these in a test script.
@kacy
kacy merged commit 3f0ccad into main Aug 3, 2026
2 checks passed
@kacy
kacy deleted the testing-assertions branch August 3, 2026 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant