Skip to content

twister: negative total number of test cases #81128

@golowanow

Description

@golowanow

Describe the bug
After the recent series of Twister status improvements (#77480 #78716) the summary report can have negative total numbers of executed test cases (see an example below).
The issue happens when a Ztest Test Suite is executed on the target platform running several test threads in parallel, so the "duplicate" log entries are counted twice at Twister whereas it collects statistics for only one Test Instance with its single set of TestCases.

This problem, which is now visible, has it's root is Twister/Ztest incomplete integration (#80706) and this issue is created to address this particular aspect with a quick fix meanwhile just to avoid negative values in Twister summary output.

To Reproduce
for example
./scripts/twister -p native_sim -T tests/drivers/build_all/sensor -j 1 -vv -ll DEBUG

Expected behavior
Correct total numbers without negative values.

Impact
Confusing Twister summary output.

Logs and console output

twister - DEBUG - run test: native_sim/native/tests/drivers/build_all/sensor/sensors.generic_test
...
INFO    - 8 test scenarios (8 test instances) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
--------------------------------------------------
Total test suites:     8
Processed test suites: 8
├─ Filtered test suites (static):       0
└─ Completed test suites:               8
   ├─ Filtered test suites (at runtime):   0
   ├─ Passed test suites:                  1
   ├─ Built only test suites:              7
   ├─ Failed test suites:                  0
   └─ Errors in test suites:               0

Filtered test suites: 0
├─ Filtered test suites (static):       0
└─ Filtered test suites (at runtime):   0
----------------------      ----------------------
Total test cases: 204
├─ Filtered test cases:   0
├─ Skipped test cases:  382
└─ Executed test cases: -178
   ├─ Passed test cases:          6
   ├─ Built only test cases:      7
   ├─ Blocked test cases:         0
   ├─ Failed test cases:          0
   └─ Errors in test cases:       0
--------------------------------------------------
INFO    - 1 of 8 executed test configurations passed (12.50%), 7 built (not run), 0 failed, 0 errored, with no warnings in 179.47 seconds.
INFO    - 6 of -185 executed test cases passed (-3.24%) on 1 out of total 860 platforms (0.12%).
INFO    - 382 selected test cases not executed: 382 skipped, 7 not run (built only).
INFO    - 1 test configurations executed on platforms, 7 test configurations were only built.

Environment (please complete the following information):

  • OS: Linux
  • Toolchain Zephyr SDK
  • Commit SHA or Version used - v4.0 rc2 currently at 6798064

Metadata

Metadata

Assignees

Labels

area: TwisterTwisterbugThe issue is a bug, or the PR is fixing a bugpriority: mediumMedium impact/importance bug

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions