fix(reports): every report format was empty under --parallel - #1006
Merged
Conversation
Closes #1004. --report-junit, --report-tap, --report-json, --report-html and --log-junit all produced a report of zero tests whenever the run was parallel. The tests ran and passed; only the report was empty. ./bashunit --report-junit r.xml tests/unit/util tests="49" ./bashunit --parallel --report-junit r.xml tests/unit/util tests="0" That is worse than an obviously broken file. --parallel is the mode people reach for in CI and junit is what CI consumes, and a junit file reporting no tests renders as a green, empty run in most systems. A suite could be fully green, fully reported, and communicating nothing. bashunit::reports::add_test is called from runner/exec.sh, which under --parallel runs inside the per-test worker. The arrays it appends to died with the worker, and state/parallel.sh -- which rebuilds counters from the .result files -- had no knowledge of them. Counters crossed the fork boundary; report rows did not. Each row is now also spooled to a run-scoped file, the same mechanism --snapshot-report-unused already uses to cross that boundary, and replayed in the parent immediately before the writers run. Fields are base64-encoded because a failure message carries newlines and arbitrary text. The arrays are still filled in add_test rather than skipped when parallel. The first version returned early there, which was wrong: the reports unit tests call add_test directly in the parent, and 32 of them started asserting against arrays nothing had touched. The parent never reaches that path for a real parallel test, so replaying the spool cannot double-count. Verified by generating all three text reports from the same fixture both ways and comparing: identical for junit, and identical in content for tap and json -- they differ only in row order, which is inherent to parallel execution. Statuses, durations, assertion counts and the multi-line failure message all survive. The new acceptance test runs the same suite sequentially and in parallel and compares the counts, so the two cannot drift apart again. 1735 sequential / 1694 parallel; fork budget unchanged.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1004
--report-junit,--report-tap,--report-json,--report-htmland--log-junitall produced a report of zero tests whenever the run was parallel. The tests ran and passed; only the report was empty.That's worse than an obviously broken file.
--parallelis the mode people reach for in CI, junit is what CI consumes, and a junit file reporting no tests renders as a green, empty run. A suite could be fully green, fully reported, and communicating nothing.🔍 Cause
reports::add_testis called fromrunner/exec.sh, which under--parallelruns inside the per-test worker. The arrays it appends to died with the worker, andstate/parallel.sh— which rebuilds counters from the.resultfiles — had no knowledge of them. Counters crossed the fork boundary; report rows did not.💡 Fix
Each row is also spooled to a run-scoped file — the same mechanism
--snapshot-report-unusedalready uses to cross that boundary — and replayed in the parent immediately before the writers run. Fields are base64-encoded because a failure message carries newlines and arbitrary text.The arrays are still filled rather than skipped when parallel. My first version returned early there, and 32 reports unit tests started failing — they call
add_testdirectly in the parent and assert the arrays. The parent never reaches that path for a real parallel test, so replaying the spool cannot double-count.✅ Verification
All three text reports generated from one fixture both ways and compared: junit identical; tap and json identical in content, differing only in row order, which is inherent to parallel execution. Statuses, durations, assertion counts and the multi-line failure message all survive.
The new acceptance test runs the same suite both ways and compares counts, so they cannot drift apart again.
make sa·make lint·bash build.sh bin -v→✅ Build verified ✅· fork budget unchanged · 1735 sequential / 1694 parallel-simple-strict.