Use a non-zero exit code if the tests failed#20878
Use a non-zero exit code if the tests failed#20878timvandermeij merged 1 commit intomozilla:masterfrom
Conversation
The bots currently detect if the tests failed or not by checking the output text, but this is error-prone as it would break if the text gets changed (which is rather unexpected) and it's non-standard as usually processes report success/failure via their exit code. This commit makes sure that the test process fails with a non-zero exit code if the tests failed, and a zero exit code otherwise. Note that we keep the same output text as before, so this change should not impact the bots, but it does form another step to unlock usage in GitHub Actions (which uses the exit code of commands to determine the status of the workflow run).
timvandermeij
force-pushed
the
exit-code
branch
from
e06b9e7 to
c798f88
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #20878 +/- ##
==========================================
+ Coverage 62.51% 62.54% +0.03%
==========================================
Files 173 173
Lines 121246 121246
==========================================
+ Hits 75796 75833 +37
+ Misses 45450 45413 -37
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
/botio test |
From: Bot.io (Windows)ReceivedCommand cmd_test from @timvandermeij received. Current queue size: 1 Live output at: http://54.193.163.58:8877/228da838fee543f/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @timvandermeij received. Current queue size: 1 Live output at: http://54.241.84.105:8877/dd6903b8b2c85c7/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/dd6903b8b2c85c7/output.txt Total script time: 45.34 mins
Image differences available at: http://54.241.84.105:8877/dd6903b8b2c85c7/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/228da838fee543f/output.txt Total script time: 70.30 mins
Image differences available at: http://54.193.163.58:8877/228da838fee543f/reftest-analyzer.html#web=eq.log |
4 participants
The bots currently detect if the tests failed or not by checking the output text, but this is error-prone as it would break if the text gets changed (which is rather unexpected) and it's non-standard as usually processes report success/failure via their exit code.
This commit makes sure that the test process fails with a non-zero exit code if the tests failed, and a zero exit code otherwise. Note that we keep the same output text as before, so this change should not impact the bots, but it does form another step to unlock usage in GitHub Actions (which uses the exit code of commands to determine the status of the workflow run).
Related to #11851.