Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twister: testcases skipped by ztest_test_skip() have reason "Unknown" in report #42414

Closed
gopiotr opened this issue Feb 2, 2022 · 0 comments · Fixed by #42416
Closed

twister: testcases skipped by ztest_test_skip() have reason "Unknown" in report #42414

gopiotr opened this issue Feb 2, 2022 · 0 comments · Fixed by #42416
Assignees
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug

Comments

@gopiotr
Copy link
Collaborator

gopiotr commented Feb 2, 2022

Describe the bug
When testcase written in ztest framework is skipped by ztest_test_skip() function, next it is placed in report with status "Unknown". For example in source code of test: zephyr/tests/kernel/common/src/main.c it can be observed, that testcase test_bitfield will be skipped on ARM boards:

#ifdef CONFIG_ARM
void test_bitfield(void)
{
	ztest_test_skip();
}
#else
extern void test_bitfield(void);
#endif

As result after perform this test on some ARM board (for example nrf52840dk_nrf52840, it can be observed following entry in twister_report.xml or twister.json:

<testcase classname="kernel.common" name="kernel.common.bitfield" time="7.675848">
<skipped type="skipped" message="Unknown"/>
</testcase>
{
    "testcase":"kernel.common.bitfield",
    "arch":"arm",
    "platform":"nrf52840dk_nrf52840",
    "ram_size":29433,
    "rom_size":122406,
    "status":"skipped",
    "reason":"Unknown"
},

To Reproduce
Steps to reproduce the behavior:

  1. Run command:
    scripts/twister --device-testing --device-serial /dev/ttyACM0 --json-report -p nrf52840dk_nrf52840 -T tests/kernel/common/
  2. Observe results in twister_report.xml or twister.json reports, especially entry for testcase kernel.common.bitfield

Expected behavior
Reason of performed skip should be more readable, because in this situation it is known reason of this skip, for example it should be like this:

<testcase classname="kernel.common" name="kernel.common.bitfield" time="7.675848">
<skipped type="skipped" message="Testcase skipped by ztest_test_skip() function in test's source code."/>
</testcase>

Impact
Report with such skip is suspicious, because it is not clear if this skip is performed as intentional skip in test's source code or if it is made by mistake/bug.

Environment (please complete the following information):

  • OS: Linux
  • Toolchain: zephyr-sdk-0.13.1
  • Commit SHA: efa170f

Additional context
I will crate PR with suitable fix soon.

@gopiotr gopiotr added bug The issue is a bug, or the PR is fixing a bug area: Twister Twister labels Feb 2, 2022
gopiotr added a commit to gopiotr/zephyr that referenced this issue Feb 3, 2022
When testcase written in ztest is skipped by ztest_test_skip() function,
then "SKIP" information is printed in output data. This "SKIP" result is
set as result of performed testcase, but the information about reason is
still "Unknown". This patch fix this problem and if all test scenario
has state "passed" (so testcases have result "PASS" or "SKIP"), reason
is still set as "Unknown" and if any of testcase has result "SKIP" then
set reason as "ztest skip".

Fixes zephyrproject-rtos#42414

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
@dkalowsk dkalowsk added the priority: low Low impact/importance bug label Feb 8, 2022
nashif pushed a commit that referenced this issue Feb 10, 2022
When testcase written in ztest is skipped by ztest_test_skip() function,
then "SKIP" information is printed in output data. This "SKIP" result is
set as result of performed testcase, but the information about reason is
still "Unknown". This patch fix this problem and if all test scenario
has state "passed" (so testcases have result "PASS" or "SKIP"), reason
is still set as "Unknown" and if any of testcase has result "SKIP" then
set reason as "ztest skip".

Fixes #42414

Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants