Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

twister: pytest: unstable pytest tests for shell #58747

Closed
PerMac opened this issue Jun 1, 2023 · 0 comments · Fixed by #58491
Closed

twister: pytest: unstable pytest tests for shell #58747

PerMac opened this issue Jun 1, 2023 · 0 comments · Fixed by #58491
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug

Comments

@PerMac
Copy link
Member

PerMac commented Jun 1, 2023

Describe the bug
A newly introduced pytest test for shell using new integration of pytest with twister (#57117) is instable. It happened on some unrelated PRs that the test was still failing on qemu after 3 retries, and the whole subset had to be restarted (e.g. https://github.com/zephyrproject-rtos/zephyr/actions/runs/5143039574/jobs/9257589092).
We also found, that our hw platforms using TFM (e.g. nrf9160dk_nrf9160_ns) are failing the test due to too tight window for the test initialization.

To Reproduce
Steps to reproduce the behavior:
On hw:
scripts/twister -T samples/subsys/testsuite/pytest/shell/ -v --platform nrf9160dk_nrf9160_ns --device-testing --device-serial /dev/ttyACM0
On qemu (not always failing):
scripts/twister -T samples/subsys/testsuite/pytest/shell/ -v ---platform qemu_cortex_m3

Expected behavior
The test is stable.

Impact
Strong annoyance. Can fail unrelated PRs and then requir a manual rerun.

Logs and console output

INFO    - 567/567 qemu_cortex_m3            samples/subsys/testsuite/pytest/shell/sample.pytest.shell  FAILED Pytest failed (qemu 6.453s)
INFO    - /__w/zephyr/zephyr/twister-out/qemu_cortex_m3/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/handler.log
ERROR   - 
==== Logging started at 2023-06-01 10:21:31.1[684](https://github.com/zephyrproject-rtos/zephyr/actions/runs/5143039574/jobs/9257589092#step:13:685)42 ====

==== Logging started at 2023-06-01 10:21:35.748572 ====

==== Logging started at 2023-06-01 10:26:14.924872 ====

==== Logging started at 2023-06-01 10:26:18.156371 ====

==== Logging started at 2023-06-01 10:27:28.320615 ====

==== Logging started at 2023-06-01 10:27:31.551860 ====

==== Logging started at 2023-06-01 10:28:41.716202 ====

==== Logging started at 2023-06-01 10:28:44.947695 ====

INFO    - /__w/zephyr/zephyr/twister-out/qemu_cortex_m3/samples/subsys/testsuite/pytest/shell/sample.pytest.shell/handler.log

Environment (please complete the following information):

  • Not related
@PerMac PerMac added the bug The issue is a bug, or the PR is fixing a bug label Jun 1, 2023
gchwier added a commit to gchwier/zephyr that referenced this issue Jun 1, 2023
@gchwier
twister: pytest: Wrap iter_stdout
9017001 
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: zephyrproject-rtos#58747

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
@gchwier gchwier mentioned this issue Jun 1, 2023
Merged
@nashif nashif added area: Twister Twister priority: medium Medium impact/importance bug labels Jun 2, 2023
@nashif nashif changed the title twister: pytest: Instable pytest tests for shell twister: pytest: unstable pytest tests for shell Jun 2, 2023
nashif pushed a commit that referenced this issue Jun 2, 2023
@gchwier @nashif
twister: pytest: Wrap iter_stdout
ab5b48b 
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: #58747

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
hlord2000 pushed a commit to hlord2000/zephyr that referenced this issue Jul 3, 2023
@gchwier @hlord2000
twister: pytest: Wrap iter_stdout
3278434 
Added waiting for prompt instead of calling time.sleep,
to fix issue with longer setting up platforms with TF-M
enabled. Wrapped iter_stdout generator, to avoid issues
when generator is created and called multiple times
in test scenarios.
Fixes: zephyrproject-rtos#58747

Signed-off-by: Grzegorz Chwierut <grzegorz.chwierut@nordicsemi.no>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: medium Medium impact/importance bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

twister: pytest: Wrap iter_stdout gchwier/zephyr
2 participants
@nashif @PerMac