🐛 Fix exit code 0 even though tests fail#1171
Merged
mkarlesky merged 3 commits intoJun 30, 2026
Merged
Conversation
Move test failure detection into post_test_fixture_execute so exit codes are correct regardless of plugin configuration. Fix a :result/:results key mismatch that was preventing result counts from reaching that hook. Resolves: ThrowTheSwitch#1112
Deduplicate failure banner entries surfaced by fix for ThrowTheSwitch#1112. A cleaner fix would split run_report's dual responsibilities, but that is a breaking plugin API change. See also: ThrowTheSwitch#1112
Contributor
Author
|
My thinking here is that |
Member
|
This is good work, @corytodd. Thank you! |
Member
|
Alas, the plugin architecture has always been a bit rickety. This is more evidence that it needs a proper rethink (coincident with finishing replacing Rake entirely). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🍍
Summary
Fix exit code always returning 0 on test failure when no reporting plugin is active.
#1112: Exit code is determined by whether an active reporting plugin passed a block to run_report returning a
non-empty failure string. With no stdout plugin configured, register_build_failure was never called and the build
silently exited 0. Fix moves failure detection into post_test_fixture_execute, which fires regardless of plugin configuration.
process_and_write_resultsreturned:result(singular) butgenerator.rbread it as:results, so result counts werealways
nilatpost_test_fixture_execute. Fixed at the source ingenerator_test_results.rbfor consistency with the restof the codebase.
Workaround
Active reporting plugins still register failures via their block, producing duplicate banner entries
alongside the core registration. Deduplicated in
print_plugin_failureswith.uniq. A cleaner fix would splitrun_report'sdual responsibilities but that is a breaking plugin API change.
Testing
Verified against examples/temp_sensor with a manually broken assertion in
TestMain.cand the removal ofreport_tests_pretty_stdoutfrom enabled plugins.With
report_tests_pretty_stdoutenabled, we still see exit 1 just as before and there are no duplicated banner messages. All new and current tests are passing . Robucop doesn't seem offended by anything in this patch set.Resolves #1112