Add pytest plugin #57117

gopiotr · 2023-04-21T10:29:05Z

In this PR we added pytest plugin into Zephyr source code. It makes possibility to create and run more "advanced" tests, which require communication with flashed application (on hardware or simulator). Prepared shell test was made only to show basic pytest plugin functionality.

To test:

Run exemplary shell test on native_posix and qemu:

./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix -p qemu_x86

Or run it on hardware:

./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p nrf52840dk_nrf52840 --device-testing --device-serial /dev/ttyACM0

expected output (on native_posix):

DEBUG   - run test: native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell
DEBUG   - Running pytest command: export PYTHONPATH=/home/redbeard/zephyrproject/zephyr/scripts/pylib/pytest-twister-ext/src:${PYTHONPATH} && pytest --twister-ext -s -q /home/redbeard/zephyrproject/zephyr/samples/subsys/testsuite/pytest/shell/pytest --build-dir=/home/redbeard/zephyrproject/zephyr/twister-out/native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell --junit-xml=/home/redbeard/zephyrproject/zephyr/twister-out/native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/report.xml --log-level=DEBUG --device-type=native -p twister_ext.plugin
DEBUG   - PYTEST: 19:19:07.192:DEBUG: Get device type "native"
DEBUG   - PYTEST: 19:19:07.193:INFO: Running command: /home/redbeard/zephyrproject/zephyr/twister-out/native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.exe
DEBUG   - PYTEST: 19:19:07.194:DEBUG: Using selector: EpollSelector
DEBUG   - PYTEST: 19:19:07.197:DEBUG: Started subprocess with PID 160766
DEBUG   - PYTEST: 19:19:08.296:DEBUG: #: 
DEBUG   - PYTEST: 19:19:08.315:DEBUG: #: uart:~$ help
DEBUG   - PYTEST: 19:19:08.316:DEBUG: #: Please press the <Tab> button to see all available commands.
DEBUG   - PYTEST: .19:19:08.440:DEBUG: Stopping all running processes for PID 160766
DEBUG   - PYTEST: 19:19:08.454:INFO: Running simulation finished with return code 0
DEBUG   - PYTEST: 19:19:08.455:DEBUG: Get device type "native"
DEBUG   - PYTEST: 19:19:08.456:INFO: Running command: /home/redbeard/zephyrproject/zephyr/twister-out/native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.exe
DEBUG   - PYTEST: 19:19:08.456:DEBUG: Using selector: EpollSelector
DEBUG   - PYTEST: 19:19:08.459:DEBUG: Started subprocess with PID 160774
DEBUG   - PYTEST: 19:19:09.559:DEBUG: #: 
DEBUG   - PYTEST: 19:19:09.581:DEBUG: #: uart:~$ kernel version
DEBUG   - PYTEST: 19:19:09.581:DEBUG: #: Zephyr version 3.3.99
DEBUG   - PYTEST: .19:19:09.682:DEBUG: Stopping all running processes for PID 160774
DEBUG   - PYTEST: 19:19:09.688:INFO: Running simulation finished with return code 0
DEBUG   - PYTEST: - generated xml file: /home/redbeard/zephyrproject/zephyr/twister-out/native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/report.xml -
DEBUG   - PYTEST: 2 passed in 2.50s
DEBUG   - Pytest cases passed
DEBUG   - run status: native_posix/samples/subsys/testsuite/pytest/shell/sample.pytest.shell passed
INFO    - 1/1 native_posix              samples/subsys/testsuite/pytest/shell/sample.pytest.shell PASSED (native 2.510s)

This PR consists of 3 parts:

pytest-twister-ext plugin

New pytest plugin was added as a standalone python package into scripts/pylib/pytest-twister-ext.

Plugin has following structure:

/scr/twister-ext/ - folder which consists all plugin source code, especially:
/src/twister_ext/plugin.py - file which contains plugin API (available options like --build-dir, --device-type, etc.).
/scr/twister-ext/device/ - there are defined device adapter classes (dedicated for hardware, native_posix and QEMU).
/scr/twister-ext/fixtures/ - at this moment there is placed one fixture dut which can be used in pytest test. It allow to automatically configure and flash/run application on device and return DeviceAbstract type object which can be used later in test for communication with device. There is plan to add there more advanced fixtures in the future.

/tests/ - unit tests dedicated for plugin - at this moment the coverage rate is around 40% - there is plan to increase it in the future.

New pytest plugin was added as a standalone "python package", because it is required by pytest to make it possible to share fixtures, classes and methods among various tests. This plugin can be used without installing it - when Twister try to use this plugin at first it exports PYTHONPATH with path to this plugin, and then during call pytest it appends to command -p twister_ext.plugin argument. This allows to use this plugin installation it by pip.

Twister modifications

To make it possible to use effectively new pytest plugin it was necessary to make some modifications to the source code of Twister. When harness: pytest is chose in sample.yaml (or testcase.yaml) file, then Twister doesn't perform flashing hardware/running simulation actions, but instead of this it call pytest and pass to it all necessary information (like path to built application or port number of connected hardware). Then, flashing (or running) application is performed in pytest test by calling dut fixture.

There is still possibility to run "old" pytest tests by Twister - (for example this one: https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/subsys/testsuite/pytest). The main difference is that now (with changes made in this PR) all pytest tests start with unflashed device. In entire Zephyr project we don't have hardware tests basing on pytest (beyond this one mentioned exemplary test), so it is hard to us to say how much this change could affect to some external project.

To summarize:
If you already use Twister & pytest in your own project, and you expect to have flashed device before pytest test start, please add dut fixture call in your test function definition.
If it's a big problem, we can consider to add something like harness: pytest_old option to maintain full backward compatibility.

Exemplary pytest shell test

Old exemplary pytest test was move from samples/subsys/testsuite/pytest/ to samples/subsys/testsuite/pytest/basic directory.
New exemplary pytest shell test was added to samples/subsys/testsuite/pytest/shell. After call Twister by ./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix, basic shell application will be built, and then Twister will call pytest with all necessary options. At this moment there are defined two pytest tests in test_shell.py: test_shell_print_help and test_shell_print_version. Both tests use dut fixture (in their definition) and thanks to this connection with hardware/simulator is established automatically and then it can be used to communicate with device via methods like write or iter_stdout. First test sends to device help command and wait for expected output, second one sends kernel version command and wait for information about used kernel version.
All output received form device is saved to $BUILD_DIR/handler.log file (the same as it done by Twister for "regular" tests).

At this moment this test works properly on native_posix, QEMU (tested on qemu_cortex_m3, qemu_cortex_a53, qemu_riscv32, qemu_leon3 and qemu_x86) and real hardware (tested on nrf52840dk_nrf52840 and other nrf "family" boards).

Known issues

To make it possible to merge this PR into the main, at least following issues should be determined/added:

Determine how new pytest plugin should be installed ~~(indirectly via scripts/requirements-run-test.txt or directly by call pip install ...)~~. Now it is not necessary to install plugin by pip to use it and run pytest tests.
To make CI tests green - update CI workflow files (.github/workflows/twister.yaml and .github/workflows/clang.yaml) to make it possible to install new pytest plugin and thus make it available in pytest tests which currently fail. Now it is not necessary to install plugin by pip to use it and run pytest tests, so .github/workflows/*.yaml don't need to be updated.
Add minimal documentation to describe how new pytest plugin works and how it can be used in tests.
~~Update CODEOWNERS and MAINTAINERS.yml files.~~ Add pytest plugin #57117 (comment)
Add to .github/workflows/twister_tests.yml CI workflow unit tests dedicated to new pytest plugin.

Other known issues which will be added/developed in the next PRs, when this PR will be merged:

At this moment Twister options like --pre-script, --post-flash-script and --post-script are not handled by pytest plugin during flashing/running application. We decided that this functionality is not "fundamental" and could be added with next PRs.
It should be determined if some part of code could be shared somehow between Twister and pytest plugin to avoid code duplication.
Currently when Twister call pytest, then it reports in final report (xml or json) whole pytest call as one test. It would be nice to make it possible to add to final report also pytest testcases (the same as Ztest testcases are added).
At this moment device adapters in pytest plugin provide iter_stdout method to read something from device. In some cases it is not the most convenient way, and it should be determined how to do it in better way (for example replace it by simple read function with byte numbers and timeout arguments).

Tracking issue: #58288

This PR don't focus on testing shell applications, but provides support for such a kind of tests, so I attach following issue:
Fixes: #52889

This PR improves integration with pytest and thanks to this it should be easier to write tests, which communicate with external programs:
Fixes: #27956

nashif · 2023-04-21T11:31:40Z

does not seem to work with qemu...

getting:

Can't access report.xml
Pytest cases failed

Output from pytest:
Traceback (most recent call last):
  File "/home/nashif/.local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main
    code = main()
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 145, in main
    config = _prepareconfig(args, plugins)
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/home/nashif/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/nashif/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/nashif/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/home/nashif/.local/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/nashif/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1016, in pytest_cmdline_parse
    self.parse(args)
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1304, in parse
    self._preparse(args, addopts=addopts)
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1190, in _preparse
    self.known_args_namespace = self._parser.parse_known_args(
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/argparsing.py", line 152, in parse_known_args
    return self.parse_known_and_unknown_args(args, namespace=namespace)[0]
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/argparsing.py", line 161, in parse_known_and_unknown_args
    optparser = self._getparser()
  File "/home/nashif/.local/lib/python3.10/site-packages/_pytest/config/argparsing.py", line 128, in _getparser
    arggroup.add_argument(*n, **a)
  File "/usr/lib/python3.10/argparse.py", line 1453, in add_argument
    return self._add_action(action)
  File "/usr/lib/python3.10/argparse.py", line 1655, in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
  File "/usr/lib/python3.10/argparse.py", line 1467, in _add_action
    self._check_conflict(action)
  File "/usr/lib/python3.10/argparse.py", line 1604, in _check_conflict
    conflict_handler(action, confl_optionals)
  File "/usr/lib/python3.10/argparse.py", line 1613, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument -O/--outdir: conflicting option strings: -O, --outdir

gopiotr · 2023-04-21T12:44:51Z

does not seem to work with qemu...

@nashif yes, I know about it - this is initial version of final PR, and there is a plan to add handling for QEMU and other hardware. At this moment it works only with big limitations. I wrote better PR description to avoid confusion. I'll inform you when it will be ready for review ;)

gchwier · 2023-05-04T13:07:32Z

Added support for qemu and native_posix.
A new test sample can be found in samples/subsys/testsuite/pytest/shell.

To install / update pytest-twister-ext plugin, use:
pip install -e scripts/pylib/pytest-twister-ext

One can test plugin with commands:
run qemu and native_posix
./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix -p qemu_x86

run on device with west-flash options:
./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p nrf9160dk_nrf9160 --device-testing --device-serial /dev/ttyACM0 --west-flash='--erase,--dev-id=xxxxxxx'

or with hardware map:
./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv --device-testing --hardware-map ~/tmp/hardware_map.yml

also added unit tests of pytest-twister-ext plugin, to run it use:
pytest -s -vv scripts/pylib/pytest-twister-ext/tests

teburd

twister -p qemu_x86_64 -T samples/subsys/testsuite/pytest

~~Fails for me without any cause, maybe I'm missing something important but this just doesn't seem to work for me.~~

Ok works fine after pip install -e scripts/pylib/pytest-twister-ext

scripts/requirements-run-test.txt

PerMac · 2023-05-25T20:21:45Z

@nashif Tracking issue: #58288

nashif · 2023-05-26T00:46:11Z

It appears that we are flashing twice? Something is wrong here. I see the device being flashed twice by observing the device LEDs and it is seen here in the log as well.

DEBUG   - Running pytest command: export PYTHONPATH=/home/nashif/zephyrproject/zephyr/scripts/pylib/pytest-twister-harness/src:${PYTHONPATH} && pytest --twister-harness -s -q /home/nashif/zephyrproject/zephyr/samples/subsys/testsuite/pytest/shell/pytest --build-dir=/home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell --junit-xml=/home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/report.xml --log-level=DEBUG --device-type=hardware --device-serial=/dev/ttyACM0 --device-serial-baud=115200 --runner=pyocd --device-id=0240000026334e450015400f5e0e000b4eb1000097969900 '--device-product=DAPLink CMSIS-DAP' -p twister_harness.plugin
DEBUG   - PYTEST: 20:44:07.70:DEBUG: Get device type "hardware"
DEBUG   - PYTEST: 20:44:07.70:INFO: Opening serial connection for /dev/ttyACM0
DEBUG   - PYTEST: 20:44:07.70:INFO: Flashing device 0240000026334e450015400f5e0e000b4eb1000097969900
DEBUG   - PYTEST: 20:44:07.71:INFO: Flashing command: /home/nashif/.local/bin/west flash --skip-rebuild --build-dir /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell --runner pyocd -- --board-id 0240000026334e450015400f5e0e000b4eb1000097969900
DEBUG   - PYTEST: 20:44:10.552:DEBUG: -- west flash: using runner pyocd
DEBUG   - PYTEST: WARNING: runners.pyocd: hex file (None) does not exist; falling back on .bin (/home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin). Consider enabling CONFIG_BUILD_OUTPUT_HEX.
DEBUG   - PYTEST: -- runners.pyocd: Flashing file: /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin
DEBUG   - PYTEST: 0001017 I Loading /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin [load_cmd]
DEBUG   - PYTEST: [---|---|---|---|---|---|---|---|---|----]
DEBUG   - PYTEST: [========================================]
DEBUG   - PYTEST: 0002900 I Erased 0 bytes (0 sectors), programmed 0 bytes (0 pages), skipped 57344 bytes (14 pages) at 29.78 kB/s [loader]
DEBUG   - PYTEST: 20:44:10.552:INFO: Flashing finished
DEBUG   - PYTEST: 20:44:11.556:DEBUG: #: help
DEBUG   - PYTEST: 20:44:11.561:DEBUG: #: Please press the <Tab> button to see all available commands.
DEBUG   - PYTEST: 20:44:11.569:DEBUG: #: You can also use the <Tab> button to prompt or auto-complete all commands or its subcommands.
DEBUG   - PYTEST: 20:44:11.577:DEBUG: #: You can try to call commands with <-h> or <--help> parameter for more information.
DEBUG   - PYTEST: 20:44:11.580:DEBUG: #: Shell supports following meta-keys:
DEBUG   - PYTEST: 20:44:11.583:DEBUG: #: Ctrl + (a key from: abcdefklnpuw)
DEBUG   - PYTEST: 20:44:11.585:DEBUG: #: Alt  + (a key from: bf)
DEBUG   - PYTEST: 20:44:11.590:DEBUG: #: Please refer to shell documentation for more details.
DEBUG   - PYTEST: 20:44:11.592:DEBUG: #: Available commands:
DEBUG   - PYTEST: .20:44:11.613:INFO: Closed serial connection for /dev/ttyACM0
DEBUG   - PYTEST: 20:44:11.615:DEBUG: Get device type "hardware"
DEBUG   - PYTEST: 20:44:11.615:INFO: Opening serial connection for /dev/ttyACM0
DEBUG   - PYTEST: 20:44:11.616:INFO: Flashing device 0240000026334e450015400f5e0e000b4eb1000097969900
DEBUG   - PYTEST: 20:44:11.616:INFO: Flashing command: /home/nashif/.local/bin/west flash --skip-rebuild --build-dir /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell --runner pyocd -- --board-id 0240000026334e450015400f5e0e000b4eb1000097969900
DEBUG   - PYTEST: 20:44:15.118:DEBUG: -- west flash: using runner pyocd
DEBUG   - PYTEST: WARNING: runners.pyocd: hex file (None) does not exist; falling back on .bin (/home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin). Consider enabling CONFIG_BUILD_OUTPUT_HEX.
DEBUG   - PYTEST: -- runners.pyocd: Flashing file: /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin
DEBUG   - PYTEST: 0001003 I Loading /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/zephyr/zephyr.bin [load_cmd]
DEBUG   - PYTEST: [---|---|---|---|---|---|---|---|---|----]
DEBUG   - PYTEST: [========================================]
DEBUG   - PYTEST: 0002889 I Erased 0 bytes (0 sectors), programmed 0 bytes (0 pages), skipped 57344 bytes (14 pages) at 29.74 kB/s [loader]
DEBUG   - PYTEST: 20:44:15.118:INFO: Flashing finished
DEBUG   - PYTEST: 20:44:16.123:DEBUG: #: kernel version
DEBUG   - PYTEST: 20:44:16.125:DEBUG: #: Zephyr version 3.3.99
DEBUG   - PYTEST: .20:44:16.133:INFO: Closed serial connection for /dev/ttyACM0
DEBUG   - PYTEST: - generated xml file: /home/nashif/zephyrproject/zephyr/twister-out/frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/report.xml -
DEBUG   - PYTEST: 2 passed in 9.07s
DEBUG   - Could not find a matching testcase for sample.pytest.shell
DEBUG   - Pytest cases passed
DEBUG   - run status: frdm_k64f/samples/subsys/testsuite/pytest/shell/sample.pytest.shell passed
INFO    - 1/1 frdm_k64f                 samples/subsys/testsuite/pytest/shell/sample.pytest.shell PASSED (device 9.074s)

gchwier · 2023-05-26T07:06:52Z

@nashif , there are two tests in test_shell.py and there is flashing before every test.

PerMac · 2023-05-26T07:17:41Z

@nashif However, the configuration is built only once and the image is reused. The dut fixture has a flashing in its setup. It is to support the test isolation principle. It is always possible to have more asserts in a single test if you don't want reflashing

PerMac · 2023-05-26T09:49:03Z

@nashif we plan to squash all commits from this PR and leave 3: add pytest plugin, add docs, and add example of pytest test. What do you think about it?

SeppoTakalo · 2023-05-26T11:12:10Z

This fails to run

twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix
...
DEBUG   - PYTEST: self._check_conflict(action)
DEBUG   - PYTEST: File "/usr/lib/python3.8/argparse.py", line 1551, in _check_conflict
DEBUG   - PYTEST: conflict_handler(action, confl_optionals)
DEBUG   - PYTEST: File "/usr/lib/python3.8/argparse.py", line 1560, in _handle_conflict_error
DEBUG   - PYTEST: raise ArgumentError(action, message % conflict_string)
DEBUG   - PYTEST: argparse.ArgumentError: argument --platform: conflicting option string: --platform

Am I missing something?

PerMac · 2023-05-26T11:20:33Z

@SeppoTakalo Maybe you still have twister v2 istalled in your env and it is generating args conflict?

gopiotr · 2023-05-26T11:25:06Z

This fails to run

twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix
...
DEBUG   - PYTEST: self._check_conflict(action)
DEBUG   - PYTEST: File "/usr/lib/python3.8/argparse.py", line 1551, in _check_conflict
DEBUG   - PYTEST: conflict_handler(action, confl_optionals)
DEBUG   - PYTEST: File "/usr/lib/python3.8/argparse.py", line 1560, in _handle_conflict_error
DEBUG   - PYTEST: raise ArgumentError(action, message % conflict_string)
DEBUG   - PYTEST: argparse.ArgumentError: argument --platform: conflicting option string: --platform

Am I missing something?

@SeppoTakalo probably you have installed pytest-twister package in your environment. Please uninstall it by pip uninstall pytest-twister or use virtualenv.

SeppoTakalo · 2023-05-26T11:48:45Z

My mistake..
I think I was accidentally launching some old Twister code.

I removed my Python virtual environment, installed it again, and used the ./scripts/twister instead of twister

It works now.

PerMac · 2023-05-26T11:58:41Z

@hakehuang @teburd @nashif We believe we've addressed all the comments. @teburd I think the only difference from your previous ack is that now it works by default without any installation (using installed version is even blocked without an extra flag). Can you please do a re-review?

Adding pytest plugin dedicated to running pytest tests in Zephyr project. This plugin provides a dut fixture which allows to handle bidirectional communication with the device under test. This version of plugin can be used for tests dedicated to real hardware, QEMU and native_posix simulator. Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no> Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no> Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no> Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>

Making the necessary changes to enable the new pytest plugin. By default Twister should work without the pytest-twister-harness plugin installed. To achieve this, each time Twister calls pytest, the PYTHONPATH environment variable is expanded and the `-p twister_harness.plugin` option is added to the pytest command. Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no> Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>

Adding exemplary pytest shell test to show possibilities of new pytest plugin. This test uses bidirectional communication between tester and device under test. Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>

Adding a stage to twister unit test workflow where tests for pytest-twister-harness plugin are executed. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>

Adding documentation for integration of twister with pytest plugin. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>

nashif · 2023-05-26T15:43:17Z

what was the reason for the pytest failing? ( i have seen this being rerun).

nashif · 2023-05-26T15:45:04Z

@nashif However, the configuration is built only once and the image is reused. The dut fixture has a flashing in its setup. It is to support the test isolation principle. It is always possible to have more asserts in a single test if you don't want reflashing

Ok, but I think this needs to be optional and something that the test decides on. Some tests might need that, some might not.

PerMac · 2023-05-26T16:03:38Z

what was the reason for the pytest failing? ( i have seen this being rerun).

Some instability during a communication with qemu. We think it was due to a slower qemu performance. We plan to add stress/performance tests to evaluate the stability and see how it can be improved.

gopiotr changed the title ~~Add pytest plugin~~ [DNM] Add pytest plugin Apr 21, 2023

gopiotr added the DNM This PR should not be merged (Do Not Merge) label Apr 21, 2023

gopiotr force-pushed the dev/pytest_plugin branch from 32246a6 to 8942152 Compare April 21, 2023 13:47

gopiotr force-pushed the dev/pytest_plugin branch from 8942152 to 7662f7d Compare May 4, 2023 12:35

gopiotr requested a review from PerMac May 4, 2023 13:10

gopiotr force-pushed the dev/pytest_plugin branch 5 times, most recently from 382d55b to 2aec417 Compare May 9, 2023 15:11

gopiotr marked this pull request as ready for review May 9, 2023 15:19

gopiotr requested review from nashif, mbolivar-nordic and galak as code owners May 9, 2023 15:19

zephyrbot added area: Twister Twister area: Testsuite Testsuite labels May 9, 2023

gopiotr changed the title ~~[DNM] Add pytest plugin~~ Add pytest plugin May 9, 2023

zephyrbot requested review from aaronemassey, asemjonovs, golowanow, hakehuang, jeremybettis and yperess May 9, 2023 15:20

zephyrbot assigned nashif May 9, 2023

gopiotr removed the DNM This PR should not be merged (Do Not Merge) label May 9, 2023

gopiotr requested a review from teburd May 9, 2023 15:21

teburd reviewed May 9, 2023

View reviewed changes

scripts/requirements-run-test.txt Outdated Show resolved Hide resolved

PerMac mentioned this pull request May 25, 2023

Integration between twister and pytest: tracking issue. #58288

Open

gopiotr force-pushed the dev/pytest_plugin branch 2 times, most recently from 9ad6962 to bdb6baa Compare May 26, 2023 10:30

PerMac requested review from dreiss, nashif, teburd and hakehuang May 26, 2023 11:54

gopiotr and others added 5 commits May 26, 2023 15:45

samples: testsuite: exemplary pytest shell test

abd9af1

Adding exemplary pytest shell test to show possibilities of new pytest plugin. This test uses bidirectional communication between tester and device under test. Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>

ci: twister: add unit tests for pytest plugin

5f12124

Adding a stage to twister unit test workflow where tests for pytest-twister-harness plugin are executed. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>

doc: twister: add documentation for pytest plugin

a669b35

Adding documentation for integration of twister with pytest plugin. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>

gopiotr force-pushed the dev/pytest_plugin branch from bdb6baa to a669b35 Compare May 26, 2023 14:06

nashif approved these changes May 26, 2023

View reviewed changes

teburd approved these changes May 26, 2023

View reviewed changes

nashif merged commit 1158a70 into zephyrproject-rtos:main May 26, 2023
27 checks passed

PerMac mentioned this pull request Jun 1, 2023

twister: pytest: unstable pytest tests for shell #58747

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pytest plugin #57117

Add pytest plugin #57117

gopiotr commented Apr 21, 2023 •

edited

nashif commented Apr 21, 2023

gopiotr commented Apr 21, 2023

gchwier commented May 4, 2023 •

edited

teburd left a comment •

edited

PerMac commented May 25, 2023

nashif commented May 26, 2023

gchwier commented May 26, 2023 •

edited

PerMac commented May 26, 2023

PerMac commented May 26, 2023

SeppoTakalo commented May 26, 2023

PerMac commented May 26, 2023

gopiotr commented May 26, 2023

SeppoTakalo commented May 26, 2023

PerMac commented May 26, 2023

nashif commented May 26, 2023

nashif commented May 26, 2023

PerMac commented May 26, 2023

Add pytest plugin #57117

Add pytest plugin #57117

Conversation

gopiotr commented Apr 21, 2023 • edited

pytest-twister-ext plugin

Twister modifications

Exemplary pytest shell test

Known issues

nashif commented Apr 21, 2023

gopiotr commented Apr 21, 2023

gchwier commented May 4, 2023 • edited

teburd left a comment • edited

Choose a reason for hiding this comment

PerMac commented May 25, 2023

nashif commented May 26, 2023

gchwier commented May 26, 2023 • edited

PerMac commented May 26, 2023

PerMac commented May 26, 2023

SeppoTakalo commented May 26, 2023

PerMac commented May 26, 2023

gopiotr commented May 26, 2023

SeppoTakalo commented May 26, 2023

PerMac commented May 26, 2023

nashif commented May 26, 2023

nashif commented May 26, 2023

PerMac commented May 26, 2023

gopiotr commented Apr 21, 2023 •

edited

gchwier commented May 4, 2023 •

edited

teburd left a comment •

edited

gchwier commented May 26, 2023 •

edited