Report test module failures via exit codes #2419
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
okurz
requested changes
Dec 20, 2023
okurz
reviewed
Dec 20, 2023
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
6c4fbea
to
ca42475
Compare
kraih
reviewed
Dec 20, 2023
kraih
reviewed
Dec 20, 2023
kraih
reviewed
Dec 20, 2023
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
2 times, most recently
from
0df7e17
to
a3847c7
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2419 +/- ##
==========================================
+ Coverage 95.12% 95.13% +0.01%
==========================================
Files 155 156 +1
Lines 15443 15519 +76
==========================================
+ Hits 14690 14764 +74
- Misses 753 755 +2 ☔ View full report in Codecov by Sentry. |
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
5fb18aa
to
940c786
Compare
kalikiana
reviewed
Dec 22, 2023
kalikiana
reviewed
Dec 22, 2023
okurz
reviewed
Dec 22, 2023
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
940c786
to
32f1f8c
Compare
okurz
reviewed
Dec 22, 2023
|
There are two ways here:
- do I remove the comment anole?
- do I make the dummy driver create the
serial0
file?
Please advise.
Liv Dywan ***@***.***> schrieb am Do., 4. Jan. 2024, 18:24:
… ***@***.**** commented on this pull request.
------------------------------
In t/14-isotovideo.t
<#2419 (comment)>
:
> @@ -238,7 +246,59 @@ subtest 'productdir variable relative/absolute' => sub {
unlike $log, qr/assert_screen_fail_test/, 'assert screen test not scheduled';
};
+subtest 'exit status from test results: no test scheduled' => sub {
+ # dummy isotovideo invocation
+ chdir($pool_dir);
+ path(bmwqemu::STATE_FILE)->remove if -e bmwqemu::STATE_FILE;
+ path('vars.json')->remove if -e 'vars.json';
+ # TODO: backend/baseclass.pm raises if serial0 does not exist.
The TODO is still there. Please remove it.
—
Reply to this email directly, view it on GitHub
<#2419 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABNP3R35H3BZ2MXWSIYKCTDYM3QWVAVCNFSM6AAAAABA5GAGGOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQMBUGY3DEOJQGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Either is fine by me. You're already deleting the file. Feel free to leave that and just get rid of the comment. |
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
26fc3d6
to
2ab2c57
Compare
kalikiana
approved these changes
Jan 5, 2024
|
https://github.com/os-autoinst/os-autoinst/actions/runs/7420191234/job/20191101201?pr=2419#step:6:78 |
perlpunk
reviewed
Jan 5, 2024
|
That's a surprising error, why would openqa (a consumer) block a change to os-autoinst? |
To spot accidental changes in the ABI. If openQA can't process isotovideo result all kinds of problems will occur. |
|
The workflow could be adjusted here by just stripping the version numbers from ci-packages.txt I guess, so that it would always install the latest. |
perlpunk
reviewed
Jan 5, 2024
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
b4802c6
to
267cb9d
Compare
|
Applied suggestions & squashed commits. |
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
267cb9d
to
f7072f6
Compare
|
Ping (?) |
Martchus
approved these changes
Jan 9, 2024
- Introduced a `--exit-status-from-test-results` / `-e` flag to
`isotovideo`.
It'll inspect the test results after a full run and compute the exit
code from them.
If:
- no test results exist: exit code = 100
- failed test results results exist: exit code = 101
(failed result => anything different than ok or softfail)
else
- exit code = 0
- Documented the flags in the perl pod section.
- Unit tested with an empty distribution, a fail module distribution and
a soft failure.
- X-mas gift: Uppercase `$return_code` to follow perl practices.
Co-authored-by: Oliver Kurz <okurz@suse.de>
Co-authored-by: Martchus <martchus@gmx.net>
josegomezr
force-pushed
the
add_exit_on_failed_test_module
branch
from
2ba161c
to
5eb5dfb
Compare
okurz
approved these changes
Jan 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduced a
--exit-status-from-test-results/-eflag toisotovideo.It'll inspect the test results after a full run and compute the exit code from them.
If:
Documented the flags in the perl pod section.
Unit tested with an empty distribution & a fail module distribution.
Complements poo#138416