-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add pytest plugin #57117
Add pytest plugin #57117
Conversation
does not seem to work with qemu... getting:
|
@nashif yes, I know about it - this is initial version of final PR, and there is a plan to add handling for QEMU and other hardware. At this moment it works only with big limitations. I wrote better PR description to avoid confusion. I'll inform you when it will be ready for review ;) |
32246a6
to
8942152
Compare
Added support for To install / update pytest-twister-ext plugin, use: One can test plugin with commands: run on device with west-flash options: or with hardware map: also added unit tests of pytest-twister-ext plugin, to run it use: |
382d55b
to
2aec417
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
twister -p qemu_x86_64 -T samples/subsys/testsuite/pytest
Fails for me without any cause, maybe I'm missing something important but this just doesn't seem to work for me.
Ok works fine after pip install -e scripts/pylib/pytest-twister-ext
It appears that we are flashing twice? Something is wrong here. I see the device being flashed twice by observing the device LEDs and it is seen here in the log as well.
|
@nashif , there are two tests in test_shell.py and there is flashing before every test. |
@nashif However, the configuration is built only once and the image is reused. The dut fixture has a flashing in its setup. It is to support the test isolation principle. It is always possible to have more asserts in a single test if you don't want reflashing |
@nashif we plan to squash all commits from this PR and leave 3: add pytest plugin, add docs, and add example of pytest test. What do you think about it? |
9ad6962
to
bdb6baa
Compare
This fails to run
Am I missing something? |
@SeppoTakalo Maybe you still have twister v2 istalled in your env and it is generating args conflict? |
@SeppoTakalo probably you have installed |
My mistake.. I removed my Python virtual environment, installed it again, and used the It works now. |
@hakehuang @teburd @nashif We believe we've addressed all the comments. @teburd I think the only difference from your previous ack is that now it works by default without any installation (using installed version is even blocked without an extra flag). Can you please do a re-review? |
Adding pytest plugin dedicated to running pytest tests in Zephyr project. This plugin provides a dut fixture which allows to handle bidirectional communication with the device under test. This version of plugin can be used for tests dedicated to real hardware, QEMU and native_posix simulator. Co-authored-by: Lukasz Fundakowski <lukasz.fundakowski@nordicsemi.no> Co-authored-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no> Co-authored-by: Katarzyna Giadla <katarzyna.giadla@nordicsemi.no> Signed-off-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no>
Making the necessary changes to enable the new pytest plugin. By default Twister should work without the pytest-twister-harness plugin installed. To achieve this, each time Twister calls pytest, the PYTHONPATH environment variable is expanded and the `-p twister_harness.plugin` option is added to the pytest command. Co-authored-by: Piotr Golyzniak <piotr.golyzniak@nordicsemi.no> Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding exemplary pytest shell test to show possibilities of new pytest plugin. This test uses bidirectional communication between tester and device under test. Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Adding a stage to twister unit test workflow where tests for pytest-twister-harness plugin are executed. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Adding documentation for integration of twister with pytest plugin. Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
bdb6baa
to
a669b35
Compare
what was the reason for the pytest failing? ( i have seen this being rerun). |
Ok, but I think this needs to be optional and something that the test decides on. Some tests might need that, some might not. |
Some instability during a communication with qemu. We think it was due to a slower qemu performance. We plan to add stress/performance tests to evaluate the stability and see how it can be improved. |
In this PR we added pytest plugin into Zephyr source code. It makes possibility to create and run more "advanced" tests, which require communication with flashed application (on hardware or simulator). Prepared shell test was made only to show basic pytest plugin functionality.
To test:
shell
test onnative_posix
andqemu
:expected output (on
native_posix
):This PR consists of 3 parts:
pytest-twister-ext plugin
New pytest plugin was added as a standalone python package into
scripts/pylib/pytest-twister-ext
.Plugin has following structure:
/scr/twister-ext/
- folder which consists all plugin source code, especially:/src/twister_ext/plugin.py
- file which contains plugin API (available options like--build-dir
,--device-type
, etc.)./scr/twister-ext/device/
- there are defined device adapter classes (dedicated for hardware, native_posix and QEMU)./scr/twister-ext/fixtures/
- at this moment there is placed one fixturedut
which can be used in pytest test. It allow to automatically configure and flash/run application on device and returnDeviceAbstract
type object which can be used later in test for communication with device. There is plan to add there more advanced fixtures in the future./tests/
- unit tests dedicated for plugin - at this moment the coverage rate is around 40% - there is plan to increase it in the future.New pytest plugin was added as a standalone "python package", because it is required by pytest to make it possible to share fixtures, classes and methods among various tests. This plugin can be used without installing it - when Twister try to use this plugin at first it exports
PYTHONPATH
with path to this plugin, and then during call pytest it appends to command-p twister_ext.plugin
argument. This allows to use this plugin installation it by pip.Twister modifications
To make it possible to use effectively new pytest plugin it was necessary to make some modifications to the source code of Twister. When
harness: pytest
is chose insample.yaml
(ortestcase.yaml
) file, then Twister doesn't perform flashing hardware/running simulation actions, but instead of this it call pytest and pass to it all necessary information (like path to built application or port number of connected hardware). Then, flashing (or running) application is performed in pytest test by callingdut
fixture.There is still possibility to run "old" pytest tests by Twister - (for example this one: https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/subsys/testsuite/pytest). The main difference is that now (with changes made in this PR) all pytest tests start with unflashed device. In entire Zephyr project we don't have hardware tests basing on pytest (beyond this one mentioned exemplary test), so it is hard to us to say how much this change could affect to some external project.
To summarize:
If you already use Twister & pytest in your own project, and you expect to have flashed device before pytest test start, please add
dut
fixture call in your test function definition.If it's a big problem, we can consider to add something like
harness: pytest_old
option to maintain full backward compatibility.Exemplary pytest shell test
Old exemplary pytest test was move from
samples/subsys/testsuite/pytest/
tosamples/subsys/testsuite/pytest/basic
directory.New exemplary pytest shell test was added to samples/subsys/testsuite/pytest/shell. After call Twister by
./scripts/twister -T samples/subsys/testsuite/pytest/shell -vv -p native_posix
, basic shell application will be built, and then Twister will call pytest with all necessary options. At this moment there are defined two pytest tests in test_shell.py:test_shell_print_help
andtest_shell_print_version
. Both tests usedut
fixture (in their definition) and thanks to this connection with hardware/simulator is established automatically and then it can be used to communicate with device via methods likewrite
oriter_stdout
. First test sends to devicehelp
command and wait for expected output, second one sendskernel version
command and wait for information about used kernel version.All output received form device is saved to
$BUILD_DIR/handler.log
file (the same as it done by Twister for "regular" tests).At this moment this test works properly on
native_posix
, QEMU (tested onqemu_cortex_m3
,qemu_cortex_a53
,qemu_riscv32
,qemu_leon3
andqemu_x86
) and real hardware (tested onnrf52840dk_nrf52840
and other nrf "family" boards).Known issues
To make it possible to merge this PR into the main, at least following issues should be determined/added:
(indirectly via. Now it is not necessary to install plugin by pip to use it and run pytest tests.scripts/requirements-run-test.txt
or directly by callpip install ...
)update CI workflow files (. Now it is not necessary to install plugin by pip to use it and run pytest tests, so.github/workflows/twister.yaml
and.github/workflows/clang.yaml
) to make it possible to install new pytest plugin and thus make it available in pytest tests which currently fail.github/workflows/*.yaml
don't need to be updated.UpdateAdd pytest plugin #57117 (comment)CODEOWNERS
andMAINTAINERS.yml
files..github/workflows/twister_tests.yml
CI workflow unit tests dedicated to new pytest plugin.Other known issues which will be added/developed in the next PRs, when this PR will be merged:
At this moment Twister options like--pre-script
,--post-flash-script
and--post-script
are not handled by pytest plugin during flashing/running application. We decided that this functionality is not "fundamental" and could be added with next PRs.iter_stdout
method to read something from device. In some cases it is not the most convenient way, and it should be determined how to do it in better way (for example replace it by simpleread
function with byte numbers and timeout arguments).Tracking issue: #58288
This PR don't focus on testing shell applications, but provides support for such a kind of tests, so I attach following issue:
Fixes: #52889
This PR improves integration with pytest and thanks to this it should be easier to write tests, which communicate with external programs:
Fixes: #27956