-
Notifications
You must be signed in to change notification settings - Fork 8.3k
twister: fixes to statistics and counting, reduce platform scan verbosity #81129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
twister: fixes to statistics and counting, reduce platform scan verbosity #81129
Conversation
e5f2dc4 to
152046b
Compare
When a suite is filtered, its testcases are also filtered and not skipped. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
In the summary, use "configurations" instead of instances. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
It is confusing to report filtered testcases as testcases that were selected but not exexuted. If they are filtered, then there should not be considered as selected. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Add notrun tests to the count of non-executed tests. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
This is very verbose and very long sometimes getting the way when trying to debug a problem. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Do not report status issues as errors, very confusing and developer end up looking at the wrong thing, instead, treat those as warnings and count them and report them at the end. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Built tests are not executed, change column title. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
44dc06e to
a26ad2d
Compare
Adapt tests for new changes in twister. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
We have been double counting some statuses, remove to get the stats right. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
a26ad2d to
e8d4b4b
Compare
golowanow
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove 'Trivial' label - it is not so anymore for most of 9 commits here.
|
just moving 'skipped' under 'selected' disbalancing the totals (skipped are not added to selected), eg. total 276 != 266, and selected 130 != 140 |
Count skipped cases as part of selected group in stats. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
beaaa20 to
8434acb
Compare
Rename variables leading to confusion between skipped and filtered suites. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Additional variable renames leading to confusion between skipped and filtered. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
c01ba84 to
ab556b2
Compare
suite stats were not correct, a mixup between skipped and filtered suites was leading to inconsistent numbers. This is now fixed. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
ab556b2 to
cc7e42e
Compare
LukaszMrugala
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like it's working, so I only have some aesthetical fixes.
Before and after the summary changes:
--------------------------------------------------
Total test suites: 27
Processed test suites: 27
└─Filtered test suites: 11
├─ Filtered test suites (static): 10
└─ Filtered test suites (at runtime): 1
└─ Selected test suites: 16
├─ Skipped test suites: 0
├─ Passed test suites: 13
├─ Built only test suites: 1
├─ Failed test suites: 1
└─ Errors in test suites: 1
---------------------- ----------------------
Total test cases: 176
├─ Filtered test cases: 29
└─ Selected test cases: 147
├─ Passed test cases: 119
├─ Skipped test cases: 1
├─ Built only test cases: 16
├─ Blocked test cases: 4
├─ Failed test cases: 2
├─ Errors in test cases: 0
├──── The following test case statuses should not appear in a proper execution ───
└─ Statusless test cases: 5
--------------------------------------------------
--------------------------------------------------
Total test suites: 27
Processed test suites: 27
├─ Filtered test suites: 11
│ ├─ Filtered test suites (static): 10
│ └─ Filtered test suites (at runtime): 1
└─ Selected test suites: 16
├─ Skipped test suites: 0
├─ Passed test suites: 13
├─ Built only test suites: 1
├─ Failed test suites: 1
└─ Errors in test suites: 1
---------------------- ----------------------
Total test cases: 176
├─ Filtered test cases: 29
└─ Selected test cases: 147
├─ Passed test cases: 119
├─ Skipped test cases: 1
├─ Built only test cases: 16
├─ Blocked test cases: 4
├─ Failed test cases: 2
├─ Errors in test cases: 0
├──── The following test case statuses should not appear in a proper execution ───
└─ Statusless test cases: 5
--------------------------------------------------
this manual creation of the tree is such a pain, especially when some of the tests are trying to parse it. Need to create this using python instead. |
Do not create the tree structure manually, use anytree instead. Signed-off-by: Anas Nashif <anas.nashif@intel.com>
76fb490 to
6c3a3fa
Compare
Fixes #81128