-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Description
Describe the bug
After the recent series of Twister status improvements (#77480 #78716) the summary report can have negative total numbers of executed test cases (see an example below).
The issue happens when a Ztest Test Suite is executed on the target platform running several test threads in parallel, so the "duplicate" log entries are counted twice at Twister whereas it collects statistics for only one Test Instance with its single set of TestCases.
This problem, which is now visible, has it's root is Twister/Ztest incomplete integration (#80706) and this issue is created to address this particular aspect with a quick fix meanwhile just to avoid negative values in Twister summary output.
To Reproduce
for example
./scripts/twister -p native_sim -T tests/drivers/build_all/sensor -j 1 -vv -ll DEBUG
Expected behavior
Correct total numbers without negative values.
Impact
Confusing Twister summary output.
Logs and console output
twister - DEBUG - run test: native_sim/native/tests/drivers/build_all/sensor/sensors.generic_test
...
INFO - 8 test scenarios (8 test instances) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
--------------------------------------------------
Total test suites: 8
Processed test suites: 8
├─ Filtered test suites (static): 0
└─ Completed test suites: 8
├─ Filtered test suites (at runtime): 0
├─ Passed test suites: 1
├─ Built only test suites: 7
├─ Failed test suites: 0
└─ Errors in test suites: 0
Filtered test suites: 0
├─ Filtered test suites (static): 0
└─ Filtered test suites (at runtime): 0
---------------------- ----------------------
Total test cases: 204
├─ Filtered test cases: 0
├─ Skipped test cases: 382
└─ Executed test cases: -178
├─ Passed test cases: 6
├─ Built only test cases: 7
├─ Blocked test cases: 0
├─ Failed test cases: 0
└─ Errors in test cases: 0
--------------------------------------------------
INFO - 1 of 8 executed test configurations passed (12.50%), 7 built (not run), 0 failed, 0 errored, with no warnings in 179.47 seconds.
INFO - 6 of -185 executed test cases passed (-3.24%) on 1 out of total 860 platforms (0.12%).
INFO - 382 selected test cases not executed: 382 skipped, 7 not run (built only).
INFO - 1 test configurations executed on platforms, 7 test configurations were only built.
Environment (please complete the following information):
- OS: Linux
- Toolchain Zephyr SDK
- Commit SHA or Version used - v4.0 rc2 currently at 6798064