name the case, the reason, and the difference in test failures - #643
Merged
Conversation
three defects made a table of cases unpleasant enough to be worth avoiding, which is the real reason exhaustive negative coverage does not get written. a failure did not say which case failed. check() took a name but only compared ints, and check_str only strings, so anything else fell back to assert_eq, whose report is literally named "assert_eq" — over a loop of seven vectors it named neither the row nor the parameters. check is now generic and takes the label, and check_str routes through it, so the two hand-specialized copies are one implementation. a text mismatch reported the two lengths. "got 412 chars, want 409" tells you nothing about a difference four hundred characters in; it now reports the first differing offset with a window of each side. and a rejection could not say why it was rejected. every negative test in the crypto modules is `assert(r.is_err)`, which passes just as happily when the call fails for a reason nobody intended — a malformed fixture, a renamed field, a check that ran before the one under test. assert_err_contains takes the message and the text it should mention. the module header now states the boundary that cost me a silent failure while writing this: these helpers keep their own tally, so calling them inside a colocated `test` block records into a count that block never prints and the assertion quietly stops counting. built-in assertions inside test blocks, these in a test script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
three defects in
std.testingmade a table of cases unpleasant enough to be worth avoiding — which is the actual reason exhaustive negative coverage doesn't get written, more than any missing helper.a failure didn't say which case failed.
check()took a name but compared only ints;check_stronly strings. anything else fell back toassert_eq, whose report is literally named"assert_eq"— over a loop of seven vectors it named neither the row nor the parameters, leaving two hex strings to diff by eye.checkis now generic and carries the label, andcheck_strroutes through it, so the two hand-specialized copies became one implementation.a text mismatch reported the two lengths.
got 412 chars, want 409tells you nothing about a difference four hundred characters in. it now reports the first differing offset with a window of each side:a rejection couldn't say why. every negative test in the crypto modules is
assert(r.is_err), which passes just as happily when the call fails for a reason nobody intended — a malformed fixture, a renamed field, a check that ran before the one under test.assert_err_contains(message, part)takes the error's message and the text it should mention. that one is a correctness gap rather than ergonomics: several existing negative tests would pass today if the failure moved to an unrelated cause.the boundary this uncovered, now documented
while adopting the labelled
checkinside a colocatedtestblock, the block's count silently dropped —std.testingkeeps its own tally, so calling these helpers inside atestblock records into a count that block never prints, and the assertion quietly stops counting. i caught it because a suite went from 10 tests to 8.the module header now states it: built-in assertions inside
testblocks, these in a test script that ends withdone().that also bounds what this PR buys. the labelled form helps
tests/cases-style scripts today; colocated module tests still can't share a table-driven runner, becauseassert_eqis undefined outside atestblock. that remains compiler work and is the next thing worth doing — but it is genuinely separate, and worth knowing that inline tables inside a test block work fine at one-module scale (the argon2 and jwt vector tables in #642 are written that way).what was tested
tests/cases/test_std_testing_helpers.pithexercises each new assertion, including the labelled form reporting by case name and the text diff; expected output refreshedmake run-regressions-only: 304 passed, 0 failedmake run-examples: 105 passed, 0 failed —examples/stdlib_test.pithoutput byte-identical, confirming the genericcheckis compatible with the 311 existing int call sitestests/cases/test_suite.pith(the heaviestcheckconsumer): 46 passednotes
check_*names are kept rather than deleted. removing them would mean rewriting 400+ call sites across four files for no user-visible gain; unifying the implementation behind them gets the dedup without the churn.