Skip to content

Conversation

@nashif
Copy link
Member

@nashif nashif commented Nov 8, 2024

  • twister: count filtered testcases as such, not as skipped
  • twister: use consistent language for test configurations
  • twister: do not report filtered cases in summary
  • twister: stats: not run tests are also part not executed summary
  • twister: remove verbose debug message about adding platforms

Fixes #81128

@nashif nashif added this to the v4.0.0 milestone Nov 8, 2024
@zephyrbot zephyrbot added the area: Twister Twister label Nov 8, 2024
@nashif nashif added the Trivial Changes that can be reviewed by anyone, i.e. doc changes, minor build system tweaks, etc. label Nov 8, 2024
@nashif nashif force-pushed the topic/twister/stats branch from e5f2dc4 to 152046b Compare November 8, 2024 15:19
When a suite is filtered, its testcases are also filtered and not
skipped.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In the summary, use "configurations" instead of instances.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is confusing to report filtered testcases as  testcases that were
selected but not exexuted. If they are filtered, then there should not
be considered as selected.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add notrun tests to the count of non-executed tests.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is very verbose and very long sometimes getting the way when trying
to debug a problem.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not report status issues as errors, very confusing and developer end
up looking at the wrong thing, instead, treat those as warnings and
count them and report them at the end.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Built tests are not executed, change column title.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif force-pushed the topic/twister/stats branch from 44dc06e to a26ad2d Compare November 9, 2024 20:09
Adapt tests for new changes in twister.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been double counting some statuses, remove to get the stats
right.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif force-pushed the topic/twister/stats branch from a26ad2d to e8d4b4b Compare November 9, 2024 21:12
Copy link
Member

@golowanow golowanow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove 'Trivial' label - it is not so anymore for most of 9 commits here.

@golowanow
Copy link
Member

golowanow commented Nov 11, 2024

just moving 'skipped' under 'selected' disbalancing the totals (skipped are not added to selected), eg.

Total test cases: 276
├─ Filtered test cases: 136
└─ Selected test cases: 130
   ├─ Passed test cases:       124
   ├─ Skipped test cases:       10
   ├─ Built only test cases:     0
   ├─ Blocked test cases:        0
   ├─ Failed test cases:         0
   ├─ Errors in test cases:      0
   ├──── The following test case statuses should not appear in a proper execution ───
   └─ Statusless test cases:     6

total 276 != 266, and selected 130 != 140

Count skipped cases as part of selected group in stats.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif force-pushed the topic/twister/stats branch from beaaa20 to 8434acb Compare November 11, 2024 11:28
@golowanow golowanow self-requested a review November 11, 2024 12:17
golowanow
golowanow previously approved these changes Nov 11, 2024
Rename variables leading to confusion between skipped and filtered
suites.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Additional variable renames leading to confusion between skipped and
filtered.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif removed the Trivial Changes that can be reviewed by anyone, i.e. doc changes, minor build system tweaks, etc. label Nov 11, 2024
@nashif nashif force-pushed the topic/twister/stats branch from c01ba84 to ab556b2 Compare November 11, 2024 23:41
suite stats were not correct, a mixup between skipped and filtered
suites was leading to inconsistent numbers. This is now fixed.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif force-pushed the topic/twister/stats branch from ab556b2 to cc7e42e Compare November 12, 2024 00:06
Copy link
Contributor

@LukaszMrugala LukaszMrugala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it's working, so I only have some aesthetical fixes.

Before and after the summary changes:

--------------------------------------------------
Total test suites:     27
Processed test suites: 27
└─Filtered test suites: 11
  ├─ Filtered test suites (static):       10
  └─ Filtered test suites (at runtime):    1
└─ Selected test suites:                16
   ├─ Skipped test suites:                  0
   ├─ Passed test suites:                  13
   ├─ Built only test suites:               1
   ├─ Failed test suites:                   1
   └─ Errors in test suites:                1
----------------------      ----------------------
Total test cases: 176
├─ Filtered test cases:  29
└─ Selected test cases: 147
   ├─ Passed test cases:       119
   ├─ Skipped test cases:        1
   ├─ Built only test cases:    16
   ├─ Blocked test cases:        4
   ├─ Failed test cases:         2
   ├─ Errors in test cases:      0
   ├──── The following test case statuses should not appear in a proper execution ───
   └─ Statusless test cases:     5
--------------------------------------------------
--------------------------------------------------
Total test suites:     27
Processed test suites: 27
├─ Filtered test suites:    11
│  ├─ Filtered test suites (static):       10
│  └─ Filtered test suites (at runtime):    1
└─ Selected test suites:    16
   ├─ Skipped test suites:                  0
   ├─ Passed test suites:                  13
   ├─ Built only test suites:               1
   ├─ Failed test suites:                   1
   └─ Errors in test suites:                1
----------------------      ----------------------
Total test cases: 176
├─ Filtered test cases:  29
└─ Selected test cases: 147
   ├─ Passed test cases:       119
   ├─ Skipped test cases:        1
   ├─ Built only test cases:    16
   ├─ Blocked test cases:        4
   ├─ Failed test cases:         2
   ├─ Errors in test cases:      0
   ├──── The following test case statuses should not appear in a proper execution ───
   └─ Statusless test cases:     5
--------------------------------------------------

@nashif
Copy link
Member Author

nashif commented Nov 12, 2024

Looks like it's working, so I only have some aesthetical fixes.

this manual creation of the tree is such a pain, especially when some of the tests are trying to parse it. Need to create this using python instead.

Do not create the tree structure manually, use anytree instead.

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
@nashif nashif force-pushed the topic/twister/stats branch from 76fb490 to 6c3a3fa Compare November 12, 2024 11:23
@nashif nashif dismissed LukaszMrugala’s stale review November 12, 2024 15:17

feedback addressed

@mmahadevan108 mmahadevan108 merged commit 049b243 into zephyrproject-rtos:main Nov 13, 2024
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

twister: negative total number of test cases

6 participants