Skip to content

🐛 Fix exit code 0 even though tests fail#1171

Merged
mkarlesky merged 3 commits into
ThrowTheSwitch:masterfrom
corytodd:fix-1112--exit-code-0-even-though-tests-fail
Jun 30, 2026
Merged

🐛 Fix exit code 0 even though tests fail#1171
mkarlesky merged 3 commits into
ThrowTheSwitch:masterfrom
corytodd:fix-1112--exit-code-0-even-though-tests-fail

Conversation

@corytodd

@corytodd corytodd commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

🍍

Summary

Fix exit code always returning 0 on test failure when no reporting plugin is active.

#1112: Exit code is determined by whether an active reporting plugin passed a block to run_report returning a
non-empty failure string. With no stdout plugin configured, register_build_failure was never called and the build
silently exited 0. Fix moves failure detection into post_test_fixture_execute, which fires regardless of plugin configuration.

process_and_write_results returned :result (singular) but generator.rb read it as :results, so result counts were
always nil at post_test_fixture_execute. Fixed at the source in generator_test_results.rb for consistency with the rest
of the codebase.

Workaround

Active reporting plugins still register failures via their block, producing duplicate banner entries
alongside the core registration. Deduplicated in print_plugin_failures with .uniq. A cleaner fix would split run_report's
dual responsibilities but that is a breaking plugin API change.

Testing

Verified against examples/temp_sensor with a manually broken assertion in TestMain.c and the removal of report_tests_pretty_stdout from enabled plugins.

  • Before this patch, the failed test produces a return code of 0.
  • After this patch, the failed test produces a return code of 1.

With report_tests_pretty_stdout enabled, we still see exit 1 just as before and there are no duplicated banner messages. All new and current tests are passing . Robucop doesn't seem offended by anything in this patch set.

Resolves #1112

corytodd added 3 commits June 28, 2026 11:04
Move test failure detection into post_test_fixture_execute so exit codes
are correct regardless of plugin configuration. Fix a :result/:results
key mismatch that was preventing result counts from reaching that hook.

Resolves: ThrowTheSwitch#1112
Deduplicate failure banner entries surfaced by fix for ThrowTheSwitch#1112. A cleaner
fix would split run_report's dual responsibilities, but that is a
breaking plugin API change.

See also: ThrowTheSwitch#1112
@corytodd

Copy link
Copy Markdown
Contributor Author

My thinking here is that graceful_fail should be used by plugins that want to force an exit 0. However, I'm a little worried that we may see failures for plugins or configurations that were depending on this implicit exit code discard behavior.

@corytodd corytodd changed the title Fix exit code 0 even though tests fail 🐛 Fix exit code 0 even though tests fail Jun 29, 2026
@mkarlesky

Copy link
Copy Markdown
Member

This is good work, @corytodd. Thank you!

@mkarlesky

Copy link
Copy Markdown
Member

Alas, the plugin architecture has always been a bit rickety. This is more evidence that it needs a proper rethink (coincident with finishing replacing Rake entirely).

@mkarlesky mkarlesky merged commit 7c0eb84 into ThrowTheSwitch:master Jun 30, 2026
@corytodd corytodd deleted the fix-1112--exit-code-0-even-though-tests-fail branch June 30, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exit code 0 even though tests fail

2 participants