Skip to content
Permalink
Browse files

sanitycheck help: how to spot tests that were not run and just built

There are a few reasons why sanitycheck will only build a test and not
run it: list them in the developer guide. Also lists the --options that
provide that information and update their --help message.

A couple other --help fixes.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
  • Loading branch information...
marc-hb authored and nashif committed Apr 8, 2019
1 parent d7ca903 commit e5cedcae43e4a368704df1dfb52ce2b1caa6cab6
Showing with 33 additions and 9 deletions.
  1. +18 −2 doc/guides/test/sanitycheck.rst
  2. +15 −7 scripts/sanitycheck
@@ -19,6 +19,21 @@ environment, but it does sufficient testing by building samples and
tests for different boards and different configurations to help keep the
full code tree buildable.

When using (at least) one ``-v`` option, sanitycheck's console output
shows for every test how the test is run (qemu, native_posix, etc.) or
whether the image is just built. There are a few reasons why sanitycheck
only builds a test and doesn't run it:

- The test is marked as ``build_only: true`` in its ``.yaml``
configuration file.
- The test configuration has defined a ``harness`` but you don't have
it or haven't set it up.
- You or some higher level automation invoked sanitycheck with
``--build-only``.

These also affect the outputs of ``--testcase-report`` and
``--detailed-report``, see their respective ``--help`` sections.

To run the script in the local tree, follow the steps below:

::
@@ -38,7 +53,8 @@ a simulated (QEMU) environment.

The sanitycheck script accepts the following optional arguments:

-h, --help show this help message and exit
-h, --help Show the complete and most up-to-date help message
and exit.
-p PLATFORM, --platform PLATFORM
Platform filter for testing. This option may be used
multiple times. Testcases will only be built/run on
@@ -349,7 +365,7 @@ extra_configs: <list of extra configurations>


build_only: <True|False> (default False)
If true, don't try to run the test under QEMU even if the
If true, don't try to run the test even if the
selected platform supports it.

build_on_all: <True|False> (default False)
@@ -3,6 +3,8 @@
# SPDX-License-Identifier: Apache-2.0
"""Zephyr Sanity Tests
Also check the "User and Developer Guides" at https://docs.zephyrproject.org/
This script scans for the set of unit test applications in the git
repository and attempts to execute them. By default, it tries to
build each test case on one platform per architecture, using a precedence
@@ -42,8 +44,8 @@ pairs:
when building or running the test case.
build_only: <True|False> (default False)
If true, don't try to run the test under QEMU even if the
selected platform supports it.
If true, don't try to run the test even if the selected platform
supports it.
build_on_all: <True|False> (default False)
If true, attempt to build test on all available platforms.
@@ -2723,8 +2725,8 @@ def parse_arguments():
"""
Artificially long but functional example:
$ ./scripts/sanitycheck -v \\
--testcase-root tests/ \\
--testcase-root mytests/ \\
--testcase-root tests/ztest/base \\
--testcase-root tests/kernel \\
--test tests/ztest/base/testing.ztest.verbose_0 \\
--test tests/kernel/fifo/fifo_api/kernel.fifo.poll
@@ -2776,7 +2778,7 @@ Artificially long but functional example:
test_xor_subtest.add_argument(
"-s", "--test", action="append",
help="Run only the specified test cases. These are named by "
"path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")
"<path/relative/to/Zephyr/base/section.name.in.testcase.yaml>")

test_xor_subtest.add_argument(
"--sub-test", action="append",
@@ -2794,7 +2796,9 @@ Artificially long but functional example:

parser.add_argument(
"-o", "--testcase-report",
help="Output a CSV spreadsheet containing results of the test run")
help="""Output a CSV spreadsheet containing results of the test run.
The handler_time column is left blank for tests that were only
compiled and not run.""")
parser.add_argument(
"-d", "--discard-report",
help="Output a CSV spreadsheet showing tests that were skipped "
@@ -2844,7 +2848,11 @@ Artificially long but functional example:
parser.add_argument("--detailed-report",
action="store",
metavar="FILENAME",
help="Generate a junit report with detailed testcase results.")
help="""Generate a junit report with detailed testcase results.
Unlike the CSV file produced by --testcase-report, this XML
report includes only tests which have run and none which were
merely built. If an image with multiple tests crashes early then
later tests are not accounted for either.""")

parser.add_argument("--timestamps",
action="store_true",

0 comments on commit e5cedca

Please sign in to comment.
You can’t perform that action at this time.