Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sanitycheck --build-only followed by --test-only fails #22948

Closed
galak opened this issue Feb 19, 2020 · 0 comments · Fixed by #23374
Closed

sanitycheck --build-only followed by --test-only fails #22948

galak opened this issue Feb 19, 2020 · 0 comments · Fixed by #23374
Assignees
@nashif
Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug

Comments

@galak
Copy link
Collaborator

galak commented Feb 19, 2020

Expect that the following would be equivalent:

sanitycheck -p qemu_cortex_m0 -T samples/

equivalent to:

sanitycheck --build-only -p qemu_cortex_m0 -T samples/
sanitycheck --test-only -p qemu_cortex_m0 -T samples/

However if we do --build-only followed by --test-only we get:

ERROR   - qemu_cortex_m0            samples/userspace/shared_mem/sample.kernel.memory_protection.shared_mem FAILED: Exited with 2
ERROR   - see: /home/galak/git/zephyr/sanity-out/qemu_cortex_m0/samples/userspace/shared_mem/sample.kernel.memory_protection.shared_mem/build.log
INFO    - Total complete:    1/  30   3%  skipped:    0, failed:    1
ERROR   - qemu_cortex_m0            samples/drivers/entropy/sample.drivers.entropy     FAILED: Timeout
ERROR   - see: /home/galak/git/zephyr/sanity-out/qemu_cortex_m0/samples/drivers/entropy/sample.drivers.entropy/handler.log
INFO    - Total complete:   17/  30  56%  skipped:    0, failed:    2
ERROR   - qemu_cortex_m0            samples/scheduler/metairq_dispatch/sample.metairq_dispatch FAILED: Exited with 2
ERROR   - see: /home/galak/git/zephyr/sanity-out/qemu_cortex_m0/samples/scheduler/metairq_dispatch/sample.metairq_dispatch/build.log
INFO    - Total complete:   18/  30  60%  skipped:    0, failed:    3
ERROR   - qemu_cortex_m0            samples/userspace/prod_consumer/sample.userspace.prod_consumer FAILED: Exited with 2
ERROR   - see: /home/galak/git/zephyr/sanity-out/qemu_cortex_m0/samples/userspace/prod_consumer/sample.userspace.prod_consumer/build.log
INFO    - Total complete:   19/  30  63%  skipped:    0, failed:    4
INFO    - 26 of 30 tests passed (86.67%), 4 failed, 0 skipped with 0 warnings in 62.81 seconds
INFO    - In total 756 test cases were executed on 0 out of total 229 platforms (0.00%)
@galak galak added bug The issue is a bug, or the PR is fixing a bug area: Sanitycheck Sanitycheck has been renamed to Twister priority: low Low impact/importance bug labels Feb 19, 2020
@galak galak self-assigned this Feb 19, 2020
galak added a commit to galak/zephyr that referenced this issue Feb 19, 2020
@galak
sanitycheck: support 'skipped' in 'passed' field in sanitycheck.csv
b57f2fe 
Currently the passed field is either 'True' or 'False'.  However when we
run tests we also have the option of skipping tests for various reasons.
Currently we treat 'skipped' tests as 'True'.

There are a few cases in which we load sanitycheck.csv to determine
which tests to run:

1. --only-failed
2. --load-tests
3. --test-only

For the --test-only case the fact that we don't track 'skipped' ends up
causing usage errors.  One would expect the following to be equivalent:

   sanitycheck -p qemu_cortex_m0 -T samples/

vs

   sanitycheck --build-only -p qemu_cortex_m0 -T samples/
   sanitycheck --test-only -p qemu_cortex_m0 -T samples/

But the second will fail in --test-only since various jobs that are
meant to be skipped are re-tried.

So we can easily fix the situation by adding 'skipped' as an option in
the 'passed' field in the csv.  We change --only-failed to load tests
based on the 'passed' field being 'False'.  We change --load-tests and
--test-only to skip tests with the passed field being 'skipped'.

Fixes zephyrproject-rtos#22948

Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
@galak galak mentioned this issue Feb 19, 2020
Closed
@nashif nashif assigned nashif and unassigned galak Mar 10, 2020
@nashif nashif mentioned this issue Mar 10, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nashif nashif

Labels
area: Sanitycheck Sanitycheck has been renamed to Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug
Projects
None yet
Milestone
No milestone
2 participants
@nashif @galak