twister: harness_config extended by test retries in testcase.yaml

[majunkier]

The ztest harness could be enhanced to support repeatability options, allowing users to specify these options directly within the harness configuration. By defining scenarios that include harness_config, we will unify the testing environment, making it easier to debug or keep same test scope.

Currently, Zephyr supports test repetition via command-line arguments:

west twister -x=CONFIG_ZTEST_REPEAT=y -x=CONFIG_ZTEST_TEST_REPEAT_COUNT=<NUMBER> -x=CONFIG_ZTEST_SUITE_REPEAT_COUNT=<NUMBER>

However, this approach has the following drawbacks:

• Lack of Standardization: Different users may specify different command-line options, leading to inconsistencies.

• Manual Configuration Overhead: Users must remember and manually pass these options for every test execution.

• Limited Automation Support: Running tests in CI/CD pipelines requires additional scripting to ensure consistent repetition settings.

By extending prj.conf, we allow users to define test repetition at the configuration level, ensuring that all builds include consistent test execution settings. This solution enables us to easily expand our testing scope, run stress tests, identify flaky tests, evaluate performance, and check for regressions.

Example of prj.conf :

CONFIG_ZTEST=y
CONFIG_ZTEST_SUITE_REPEAT_COUNT=3
CONFIG_ZTEST_TEST_REPEAT_COUNT=2

Users can also define test repetition in testcase.yaml:

tests:
  my.test.scenario:
    platform_allow: qemu_x86
    harness: console
    harness_config:
      ztest_suite_repeat: 2
      ztest_test_repeat: 1

[nashif]

You need to add an example/test of this being used and document this new behaviour.

[nashif]

still missing a test

[nashif]

still missing a test

please add a test into tests/ztest that uses those keywords and shows how repitition is achieved and handled, a twister unit test is not enough.

[nashif]

ztest_suite_repeat and ztest_test_repeat are configuration options of ztest harness, not console.
How is this going to work when used in other tests? does each ztest need to start parsing console and output?

[majunkier]

I used the console harness to verify that this change works as expected. I have added test for ztest harness where number of repetitions per suite or test are configured in prj.conf.

[golowanow]

From TestInstance implementation change it appears to me that suite_repeat, test_repeat, and test_shuffle are generic parameters, and ztest is going to be just one of possible implementations, but later the ztest_* properties (with different names to what is in the doc) were introduced for Harness (for any harness, actually), and I'm confused.

[majunkier]

done

[golowanow]

what about suite_shuffle ?

[majunkier]

done

[yperess]

How is this different from adding this to the test yaml?

extra_configs:
  - CONFIG_ZTEST_REPEAT=y
  - CONFIG_ZTEST_REPEAT_COUNT=<NUMBER>
  - CONFIG_ZTEST_SUITE_REPEAT_COUNT=<NUMBER>

[majunkier]

@nashif Could you share your perspective?

[nashif]

How is this different from adding this to the test yaml?

extra_configs:
  - CONFIG_ZTEST_REPEAT=y
  - CONFIG_ZTEST_REPEAT_COUNT=<NUMBER>
  - CONFIG_ZTEST_SUITE_REPEAT_COUNT=<NUMBER>

it does have the same outcome with the main difference of seperating how you are testing from what is being tested, i.e. the extra_configs are used to add new scenarios using different config options of the test subject, the ZTEST config options are not what is being tested, it is how we execute the test, so they should be part of the harness configuration.

[gbarkadiusz]

Suggested change

	* Copyright (c) 2024 Intel Corporation
	* Copyright (c) 2025 Intel Corporation

[gbarkadiusz]

LGTM