Skip to content

twister: a lot of NONE test case statuses #80801

@golowanow

Description

@golowanow

Describe the bug
Twister don't set TestCase result status (remains NONE) in two execution modes:

  • a test suite has testcase.yaml with an integration platform which is not available to run - an example observed on CI
  • --cmake-only.

To Reproduce
./scripts/twister -j 1 -vv -ll DEBUG -p sam_e70_xplained/same70q21 -T tests/subsys/dsp/basicmath
./scripts/twister -j 1 -vv -ll DEBUG -p native_sim -T tests/kernel/sleep --cmake-only

Expected behavior
TestCases have correct status, no errors.

Impact

  • incorrect (missed) TestCase result statuses,
  • flood of ERRORs in Twister log.

Logs and console output
./scripts/twister -j 1 -vv -ll DEBUG -p sam_e70_xplained/same70q21 -T tests/subsys/dsp/basicmath

ERROR - A None status detected in instance sam_e70_xplained/same70q21/tests/subsys/dsp/basicmath/zdsp.basicmath, test case zdsp.basicmath.zdsp_clip_q7_in_place_0d_40
... a lot of ....
ERROR - A None status detected in instance sam_e70_xplained/same70q21/tests/subsys/dsp/basicmath/zdsp.basicmath, test case zdsp.basicmath.zdsp_add_f16_7.
INFO - 2/2 sam_e70_xplained/same70q21 tests/subsys/dsp/basicmath/zdsp.basicmath          NOT RUN (build)
INFO -                                    zdsp.basicmath.zdsp_add_f16_7                                               NONE
... a lot of ....
INFO - 3 test scenarios (3 test instances) selected, 1 configurations filtered (1 by static filter, 0 at runtime).
--------------------------------------------------                                    
Total test suites:     3                                                              
Processed test suites: 3                                                              
├─ Filtered test suites (static):       1                                             
└─ Completed test suites:               2                                             
   ├─ Filtered test suites (at runtime):   0                                          
   ├─ Passed test suites:                  0                                          
   ├─ Built only test suites:              2                                          
   ├─ Failed test suites:                  0                                          
   └─ Errors in test suites:               0                                          

Filtered test suites: 1                                                               
├─ Filtered test suites (static):       1                                             
└─ Filtered test suites (at runtime):   0                                             
----------------------      ----------------------                                    
Total test cases: 949                                                                 
├─ Filtered test cases:   1                                                           
├─ Skipped test cases:    0                                                           
└─ Executed test cases: 948                                                           
   ├─ Passed test cases:         0                                                    
   ├─ Built only test cases:     0                                                    
   ├─ Blocked test cases:        0                                                    
   ├─ Failed test cases:         0                                                    
   ├─ Errors in test cases:      0                                                    
   ├──── The following test case statuses should not appear in a proper execution ─── 
   └─ Statusless test cases:   948                                                    
--------------------------------------------------                                    
twister - INFO - 0 of 2 executed test configurations passed (0.00%), 2 built (not run), 0 failed, 0 errored, with no warnings in 46.73 seconds.
twister - INFO - 0 of 948 executed test cases passed (0.00%), 948 without a status on 1 out of total 860 platforms (0.12%).
twister - INFO - 1 selected test cases not executed: 1 filtered.
twister - INFO - 0 test configurations executed on platforms, 2 test configurations were only built.
twister - DEBUG - zdsp.basicmath.zdsp_clip_q7_in_place_0d_40: No status
twister - DEBUG - zdsp.basicmath.zdsp_clip_q7_in_place_c0_40: No status
... a lot of ....
twister - DEBUG - zdsp.basicmath.fpu.zdsp_add_f16_7: No status
twister - INFO - Run completed

./scripts/twister -j 1 -vv -ll DEBUG -p native_sim -T tests/kernel/sleep --cmake-only

INFO    - 2 test scenarios (2 test instances) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
--------------------------------------------------
Total test suites:     2
Processed test suites: 2
├─ Filtered test suites (static):       0
└─ Completed test suites:               2
   ├─ Filtered test suites (at runtime):   0
   ├─ Passed test suites:                  2
   ├─ Built only test suites:              0
   ├─ Failed test suites:                  0
   └─ Errors in test suites:               0

Filtered test suites: 0
├─ Filtered test suites (static):       0
└─ Filtered test suites (at runtime):   0
----------------------      ----------------------
Total test cases: 6
├─ Filtered test cases: 0
├─ Skipped test cases:  0
└─ Executed test cases: 6
   ├─ Passed test cases:       0
   ├─ Built only test cases:   0
   ├─ Blocked test cases:      0
   ├─ Failed test cases:       0
   ├─ Errors in test cases:    0
   ├──── The following test case statuses should not appear in a proper execution ───
   └─ Statusless test cases:   6
--------------------------------------------------
INFO    - 2 of 2 executed test configurations passed (100.00%), 0 built (not run), 0 failed, 0 errored, with no warnings in 16.64 seconds.
INFO    - 0 of 6 executed test cases passed (0.00%), 6 without a status on 1 out of total 860 platforms (0.12%).
INFO    - 0 test configurations executed on platforms, 2 test configurations were only built.
DEBUG   - kernel.common.timing.sleep: No status
....
DEBUG   - kernel.common.timing.minimallibc.usleep: No status
INFO    - Run completed

Environment (please complete the following information):

  • OS: Linux
  • Toolchain Zephyr SDK
  • Commit SHA or Version used - v4.0.0-rc2

Introduced with #78716

Metadata

Metadata

Assignees

Labels

area: TwisterTwisterbugThe issue is a bug, or the PR is fixing a bugpriority: mediumMedium impact/importance bug

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions