twister: ztest: test execution status lost and set to NONE

[golowanow]

Describe the bug

The Twister tool integration with Ztest framework is not fully implemented yet to handle Ztest test results properly: the Ztest Suite names are not taken into account by Twister to identify the corresponding TestCase precisely, so a Ztest test's status is not assigned to the corresponding TestCase object at Twister and its status remains 'None', whereas the status is set at another TestCase which actual status value is lost, eventually the resulting total execution counters not correct as well as twister.json report.

The issue is observed in these situations:

• Ztest application with multiple test suites having same test names.
• Ztest suite is conditionally 'skipped' entirely on execution with all its tests.
• Ztest suite implemented in C++ with namespaces.

The 'in-tree' test suites known as affected by the issue (thanks to @arikgreen) :

• tests/ztest
• tests/drivers/can/api/
• tests/drivers/i2c/i2c_emul/
• tests/drivers/sensor/adltc2990
• tests/drivers/sensor/ina230
• tests/drivers/sensor/ina237
• tests/net/socket/reuseaddr_reuseport/
• tests/subsys/fs/multi-fs/

To Reproduce
Run the above mentioned test suites, for example /scripts/twister -vv -p native_sim -T ./tests/ztest/summary and check its console output and/or twister.json for test cases with "status":"None"

Expected behavior
All Ztest test cases executed get correct result statuses assigned in the Twister report and its summary totals.

Impact
Incorrect and misleading test results.

Logs and console output

/scripts/twister -p native_sim -T ./tests/ztest/summary
...
DEBUG - OUTPUT: Running TESTSUITE suite1
DEBUG - OUTPUT: ===================================================================
DEBUG - OUTPUT: START - test_foo
DEBUG - OUTPUT:  PASS - test_foo in 0.000 seconds
DEBUG - OUTPUT: ===================================================================
DEBUG - OUTPUT: TESTSUITE suite1 succeeded
DEBUG - OUTPUT: Running TESTSUITE suite2
DEBUG - OUTPUT: ===================================================================
DEBUG - OUTPUT: START - test_foo
DEBUG - OUTPUT:  SKIP - test_foo in 0.000 seconds
DEBUG - OUTPUT: ===================================================================
DEBUG - OUTPUT: TESTSUITE suite2 succeeded
DEBUG - OUTPUT:
DEBUG - OUTPUT: ------ TESTSUITE SUMMARY START ------
DEBUG - OUTPUT:
DEBUG - OUTPUT: SUITE PASS - 100.00% [suite1]: pass = 1, fail = 0, skip = 0, total = 1 duration = 0.000 seconds
DEBUG - OUTPUT:  - PASS - [suite1.test_foo] duration = 0.000 seconds
DEBUG - OUTPUT:
DEBUG - OUTPUT: SUITE SKIP -   0.00% [suite2]: pass = 0, fail = 0, skip = 1, total = 1 duration = 0.000 seconds
DEBUG - OUTPUT:  - SKIP - [suite2.test_foo] duration = 0.000 seconds
DEBUG - OUTPUT:
DEBUG - OUTPUT: ------ TESTSUITE SUMMARY END ------
DEBUG - OUTPUT:
DEBUG - OUTPUT: ===================================================================
DEBUG - OUTPUT: RunID: c6e962f4c1b5727c841b7071074c8260
DEBUG - OUTPUT: PROJECT EXECUTION SUCCESSFUL
DEBUG - Expected suite names:['suite1', 'suite2']
DEBUG - Detected suite names:['suite1', 'suite2']
DEBUG - run status: native_sim/tests/ztest/summary/testing.ztest.summary.shared_unit_test passed
...
INFO    - 1/1 native_sim/native         tests/ztest/summary/testing.ztest.summary.shared_unit_test PASSED (native 0.017s)
INFO    -                                    testing.ztest.summary.shared_unit_test.foo                                  SKIPPED      ztest skip
INFO    -                                    testing.ztest.summary.shared_unit_test.foo                                  NONE
INFO    - 1 test scenarios (1 test instances) selected, 0 configurations skipped (0 by static filter, 0 at runtime).
--------------------------------
Total test suites: 1
Total test cases: 2
Executed test cases: 1
Skipped test cases: 1
Completed test suites: 1
Passing test suites: 1
Built only test suites: 0
Failing test suites: 0
Skipped test suites: 0
Skipped test suites (runtime): 0
Skipped test suites (filter): 0
Errors: 0
--------------------------------
INFO    - 1 of 1 test configurations passed (100.00%), 0 built (not run), 0 failed, 0 errored, 0 skipped with 0 warnings in 11.61 seconds
INFO    - In total 1 test cases were executed, 1 skipped on 1 out of total 860 platforms (0.12%)
INFO    - 1 test configurations executed on platforms, 0 test configurations were only built.
...
DEBUG   - testing.ztest.summary.shared_unit_test.foo: No status

Environment (please complete the following information):

• OS: Linux
• Toolchain Zephyr SDK
• Commit SHA: 124aae3