-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Labels
area: TwisterTwisterTwisterbugThe issue is a bug, or the PR is fixing a bugThe issue is a bug, or the PR is fixing a bugpriority: mediumMedium impact/importance bugMedium impact/importance bug
Description
Describe the bug
Twister don't set TestCase result status (remains NONE) in two execution modes:
- a test suite has
testcase.yamlwith anintegration platformwhich is not available to run - an example observed on CI --cmake-only.
To Reproduce
./scripts/twister -j 1 -vv -ll DEBUG -p sam_e70_xplained/same70q21 -T tests/subsys/dsp/basicmath
./scripts/twister -j 1 -vv -ll DEBUG -p native_sim -T tests/kernel/sleep --cmake-only
Expected behavior
TestCases have correct status, no errors.
Impact
- incorrect (missed) TestCase result statuses,
- flood of ERRORs in Twister log.
Logs and console output
./scripts/twister -j 1 -vv -ll DEBUG -p sam_e70_xplained/same70q21 -T tests/subsys/dsp/basicmath
ERROR - A None status detected in instance sam_e70_xplained/same70q21/tests/subsys/dsp/basicmath/zdsp.basicmath, test case zdsp.basicmath.zdsp_clip_q7_in_place_0d_40
... a lot of ....
ERROR - A None status detected in instance sam_e70_xplained/same70q21/tests/subsys/dsp/basicmath/zdsp.basicmath, test case zdsp.basicmath.zdsp_add_f16_7.
INFO - 2/2 sam_e70_xplained/same70q21 tests/subsys/dsp/basicmath/zdsp.basicmath NOT RUN (build)
INFO - zdsp.basicmath.zdsp_add_f16_7 NONE
... a lot of ....
INFO - 3 test scenarios (3 test instances) selected, 1 configurations filtered (1 by static filter, 0 at runtime).
--------------------------------------------------
Total test suites: 3
Processed test suites: 3
├─ Filtered test suites (static): 1
└─ Completed test suites: 2
├─ Filtered test suites (at runtime): 0
├─ Passed test suites: 0
├─ Built only test suites: 2
├─ Failed test suites: 0
└─ Errors in test suites: 0
Filtered test suites: 1
├─ Filtered test suites (static): 1
└─ Filtered test suites (at runtime): 0
---------------------- ----------------------
Total test cases: 949
├─ Filtered test cases: 1
├─ Skipped test cases: 0
└─ Executed test cases: 948
├─ Passed test cases: 0
├─ Built only test cases: 0
├─ Blocked test cases: 0
├─ Failed test cases: 0
├─ Errors in test cases: 0
├──── The following test case statuses should not appear in a proper execution ───
└─ Statusless test cases: 948
--------------------------------------------------
twister - INFO - 0 of 2 executed test configurations passed (0.00%), 2 built (not run), 0 failed, 0 errored, with no warnings in 46.73 seconds.
twister - INFO - 0 of 948 executed test cases passed (0.00%), 948 without a status on 1 out of total 860 platforms (0.12%).
twister - INFO - 1 selected test cases not executed: 1 filtered.
twister - INFO - 0 test configurations executed on platforms, 2 test configurations were only built.
twister - DEBUG - zdsp.basicmath.zdsp_clip_q7_in_place_0d_40: No status
twister - DEBUG - zdsp.basicmath.zdsp_clip_q7_in_place_c0_40: No status
... a lot of ....
twister - DEBUG - zdsp.basicmath.fpu.zdsp_add_f16_7: No status
twister - INFO - Run completed
./scripts/twister -j 1 -vv -ll DEBUG -p native_sim -T tests/kernel/sleep --cmake-only
INFO - 2 test scenarios (2 test instances) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
--------------------------------------------------
Total test suites: 2
Processed test suites: 2
├─ Filtered test suites (static): 0
└─ Completed test suites: 2
├─ Filtered test suites (at runtime): 0
├─ Passed test suites: 2
├─ Built only test suites: 0
├─ Failed test suites: 0
└─ Errors in test suites: 0
Filtered test suites: 0
├─ Filtered test suites (static): 0
└─ Filtered test suites (at runtime): 0
---------------------- ----------------------
Total test cases: 6
├─ Filtered test cases: 0
├─ Skipped test cases: 0
└─ Executed test cases: 6
├─ Passed test cases: 0
├─ Built only test cases: 0
├─ Blocked test cases: 0
├─ Failed test cases: 0
├─ Errors in test cases: 0
├──── The following test case statuses should not appear in a proper execution ───
└─ Statusless test cases: 6
--------------------------------------------------
INFO - 2 of 2 executed test configurations passed (100.00%), 0 built (not run), 0 failed, 0 errored, with no warnings in 16.64 seconds.
INFO - 0 of 6 executed test cases passed (0.00%), 6 without a status on 1 out of total 860 platforms (0.12%).
INFO - 0 test configurations executed on platforms, 2 test configurations were only built.
DEBUG - kernel.common.timing.sleep: No status
....
DEBUG - kernel.common.timing.minimallibc.usleep: No status
INFO - Run completed
Environment (please complete the following information):
- OS: Linux
- Toolchain Zephyr SDK
- Commit SHA or Version used - v4.0.0-rc2
Introduced with #78716
Metadata
Metadata
Assignees
Labels
area: TwisterTwisterTwisterbugThe issue is a bug, or the PR is fixing a bugThe issue is a bug, or the PR is fixing a bugpriority: mediumMedium impact/importance bugMedium impact/importance bug