sanitycheck: Add a feature which can handle pytest script. #29427

youhuazhu · 2020-10-22T02:09:51Z

Add a class of Pytest(Harness) which can handle pytest script
in harness.py
Use running_dir to store current test directory which be
used in pytest scrpt.
Add a new section(handle_script) which used to specify the pytest
script in yaml file.
Add an example in tests/pytest_sample/

galak · 2020-10-22T02:42:56Z

What's the use case for this?

youhuazhu · 2020-10-22T02:55:01Z

What's the use case for this?

In same case， we need to excute external script for check the zephyr output data, for example, We need to convert the tracing CTF data to text format and check if it contain thread/semaphore ....
so as Anas's suggestion, I add this feature into sanitycheck for common use.

LeiW000 · 2020-10-22T05:51:47Z

@nashif please help to take look at it.

nashif · 2020-10-22T12:02:26Z

What's the use case for this?

for post-run/post-build tests, like examining recorded output (tracing), running tests on generated output and artifacts etc.

scripts/sanity_chk/harness.py

doc/guides/test/sanitycheck.rst

scripts/sanity_chk/harness.py

scripts/sanity_chk/sanitylib.py

doc/guides/test/sanitycheck.rst

scripts/sanity_chk/harness.py

youhuazhu · 2020-10-29T01:56:19Z

Hi @nashif , I have updated this PR accourding your comments, would you help to review it again?

LeiW000 · 2020-11-05T13:18:55Z

@nashif , could you take look at it again? Youhua had updated the code.

PerMac · 2020-12-11T11:58:24Z

doc/guides/test/sanitycheck.rst

+    ::
+
+        tests:
+          test:


https://docs.zephyrproject.org/latest/guides/test/sanitycheck.html#test-cases:

Each test identifier shall start with a section followed by a subsection separated by a dot. For example, a test that covers semaphores in the kernel shall start with kernel.sempahore.

You can use pytest.example in here.

done in #33227

PerMac · 2020-12-11T12:01:10Z

doc/guides/test/sanitycheck.rst

+          test:
+            harness: pytest
+            harness_config:
+              pytest_root: [pytest directory name]


Since you added the option for specifying an individual timeout for a pytest test, it would be beneficial if you add its usage in this example.

timeout removed in #33227

PerMac · 2020-12-11T12:02:50Z

scripts/sanity_chk/harness.py

+            self.running_dir = config.get('running_dir', None)
+            self.source_dir = config.get('source_dir', None)
+            self.pytest_root = config.get('pytest_root', 'pytest')
+            self.pytest_timeout = config.get('pytest_timeout', 'pytest')


copy-paste error? -> config.get('pytest_timeout', 15) ?

Fixed.
And the example commit is included in this PR

scripts/sanity_chk/harness.py

PerMac

I am not sure if the location of the pytest example is correct within the zephyr tree. In zephyr/tests/ directories correspond to different modules. I think it will be better if pytest_sample is one level deeper, eg. zephyr/tests/pytest/pytest_sample @nashif Any thoughts on this?
The output of pytest is quite verbose. By default, you should capture the stdout of pytest when called with subprocess and only show it when sanitycheck is called with -v -v
There is something wrong going on when I try this pytest example. This is the output I am getting when calling: ./scripts/twister -T tests/pytest_sample/:

show output

$ ./scripts/twister -T tests/pytest_sample/
ZEPHYR_BASE unset, using "/home/maciej/zephyrproject2/zephyr"
Renaming output directory to /home/maciej/zephyrproject2/zephyr/twister-out.42
INFO - Zephyr version: zephyr-v2.4.0-2373-g12751d994e
INFO - JOBS: 4
INFO - Selecting default platforms per test case
INFO - Building initial testcase list...
INFO - 1 test suites (2 configurations) selected, 22 configurations discarded due to filters.
INFO - Adding tasks to the queue...
INFO - Added initial list of jobs to queue
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.07s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.04s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.41s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.50s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
--------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86_64/tests/pytest_sample/pytest.sample/report.html ----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
INFO - Total complete: 1/ 2 50% skipped: 0, failed: 0================================================================================ test session starts ================================================================================
platform linux -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/maciej/zephyrproject2/zephyr
collected 1 item

tests/pytest_sample/pytest/test_ctf.py /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
/home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample
.

================================================================================= warnings summary ==================================================================================
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446
/home/maciej/.pyenv/versions/zephyr377/lib/python3.7/site-packages/_pytest/junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.
_issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------------------------- generated xml file: /home/maciej/zephyrproject2/zephyr/twister-out/qemu_x86/tests/pytest_sample/pytest.sample/report.html -----------------------------
=========================================================================== 1 passed, 1 warning in 0.01s ============================================================================
INFO - Total complete: 2/ 2 100% skipped: 0, failed: 0
INFO - 2 of 2 test configurations passed (100.00%), 0 failed, 0 skipped with 0 warnings in 7.67 seconds
INFO - In total 2 test cases were executed, 0 skipped on 2 out of total 313 platforms (0.64%)
INFO - 2 test configurations executed on platforms, 0 test configurations were only built.
INFO - Saving reports...
INFO - Writing xunit report /home/maciej/zephyrproject2/zephyr/twister-out/twister.xml...
INFO - Writing xunit report /home/maciej/zephyrproject2/zephyr/twister-out/twister_report.xml...
INFO - Writing JSON report /home/maciej/zephyrproject2/zephyr/twister-out/twister.json
INFO - Run completed

It seems that the same tests are called multiple times. I tried debugging and I think the thread where harness.handle(line) is going on is called too many times (line 988 in twisterlib.py). I belive it is some harness design issue, as from my understanding, there is no real line checking going on in this place. Other harnesses are checking lines from output if they match some given pattern. In pytest harness, the procedure not only check the lines in the output but call pytest with subprocess each time it happens. This behavior requires redesign. @nashif any hints on how this can be done?

It also looks like a weird design that pytest test requires now the full structure of standard tests in c, i.e. having in main.c

void test_pytest(void)
{
	TC_PRINT("Hello world\n");
}

where in the end the actual test is in test_ctf.py and have nothing to do with whatever is going on in main.c:

def test_case(cmdopt):
    ''' test case '''
    if not os.path.exists(cmdopt):
        assert 0
    print(cmdopt)

I guess that this is some simple approach which doesn't require changing of the default mode, where ztest structures are used. I will accept this approach, but it is pretty confusing. This is also something which would require your attention @nashif

PerMac · 2020-12-21T08:42:33Z

doc/guides/test/twister.rst

@@ -369,6 +369,11 @@ harness_config: <harness configuration options>

        Only one fixture can be defined per testcase.

+    handle_script: <pytest dirctory> (default pytest)
+        Specify a pytest directory which need to excute when test case begin to running,
+        default pytest directory name is pytest, after pytest finished, sanitycheck will


sanitycheck -> twister

updated. refine the output of pytest

@PerMac Can you check it? thank you!

1. Add a class of Pytest(Harness) which can handle pytest script in harness.py 2. Use running_dir to store current test directory which be used in pytest scrpt. 3. Add a new section(handle_script) which used to specify the pytest script in yaml file. 4. Add usage of this feature into zephyr doc. Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>

In this example, python test case get the running directory by handling the "--cmdopt" option passed by pytest. Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>

LeiW000 · 2021-03-01T03:34:22Z

@PerMac @nashif, your comments have been addressed. Could you take look at the PR again? The PR has been there for a few months.

nashif · 2021-03-06T12:55:56Z

scripts/pylib/twister/harness.py

+			'-q'
+        ]
+
+        with subprocess.Popen(cmd) as proc:


this will generate lots of noise

i9:pytest_sample((HEAD detached from upstream/pull/29427)): twister -p qemu_x86 -T . INFO - Zephyr version: zephyr-v2.5.0-838-gba4a8fc61df2 INFO - JOBS: 20 INFO - Using 'zephyr' toolchain. INFO - Building initial testcase list... INFO - 1 test scenarios (1 configurations) selected, 0 configurations discarded due to filters. INFO - Adding tasks to the queue... INFO - Added initial list of jobs to queue handle cmdopt: /home/nashif/Work/zephyrproject/zephyr/tests/pytest/pytest_sample/twister-out/qemu_x86/tests/pytest/pytest_sample/pytest.sample run test cases in: /home/nashif/Work/zephyrproject/zephyr/tests/pytest/pytest_sample/twister-out/qemu_x86/tests/pytest/pytest_sample/pytest.sample . - generated xml file: /home/nashif/Work/zephyrproject/zephyr/tests/pytest/pytest_sample/twister-out/qemu_x86/tests/pytest/pytest_sample/pytest.sample/report.html - 1 passed in 0.01s INFO - Total complete: 1/ 1 100% skipped: 0, failed: 0 INFO - 1 of 1 test configurations passed (100.00%), 0 failed, 0 skipped with 0 warnings in 19.91 seconds INFO - In total 1 test cases were executed, 0 skipped on 1 out of total 330 platforms (0.30%) INFO - 1 test configurations executed on platforms, 0 test configurations were only built. INFO - Saving reports... INFO - Writing xunit report /home/nashif/Work/zephyrproject/zephyr/tests/pytest/pytest_sample/twister-out/twister.xml... INFO - Writing xunit report /home/nashif/Work/zephyrproject/zephyr/tests/pytest/pytest_sample/twister-out/twister_report.xml... INFO - Run completed

We do not want to see pytest intermediate results in the twister output.

fixed in #33227

nashif · 2021-03-06T12:56:16Z

scripts/pylib/twister/harness.py

+			'--cmdopt',
+			self.running_dir,
+			'--junit-xml',
+			os.path.join(self.running_dir, 'report.html'),


fixed in #33227

nashif · 2021-03-06T12:57:06Z

scripts/pylib/twister/harness.py

@@ -41,6 +49,10 @@ def configure(self, instance):
            self.repeat = config.get('repeat', 1)
            self.ordered = config.get('ordered', True)
            self.record = config.get('record', {})
+            self.running_dir = config.get('running_dir', None)


why are we defining all of this, we know where we are running and where the source directory is.

fixed in #33227

nashif · 2021-03-06T12:57:32Z

scripts/pylib/twister/harness.py

+            self.running_dir = config.get('running_dir', None)
+            self.source_dir = config.get('source_dir', None)
+            self.pytest_root = config.get('pytest_root', 'pytest')
+            self.pytest_timeout = config.get('pytest_timeout', 15)


why introducing this new timeout, the test scenario already has a timeout.

removed in #33227

nashif · 2021-03-06T12:58:13Z

scripts/schemas/twister/testcase-schema.yaml

+          "pytest_root":
+            type: str
+            required: no
+          "pytest_timeout":


no need for a new timeout, use the existing one

nashif · 2021-03-06T12:59:02Z

tests/pytest/pytest_sample/pytest/test_ctf.py

@@ -0,0 +1,24 @@
+# Copyright (c) 2020 Intel Corporation.
+#
+# SPDX-License-Identifier: Apache-2.0


test_ctf? need another file name.

fixed in #33227

nashif · 2021-03-06T12:59:27Z

tests/pytest/pytest_sample/testcase.yaml

@@ -0,0 +1,4 @@
+tests:


move this under tests/testsuite/pytest

Can't push to youhua's repo any more, I update this PR to #33227

PerMac

Comments added. In general, I find the workflow for pytest test convoluted and the implementation to be suboptimal.

PerMac · 2021-03-05T12:13:18Z

scripts/pylib/twister/harness.py

+			pytest handle output as a whole, it's not necessary to handle it
+			line by line
+        '''
+        global pytest_running_dir


global? AFAIK it is highly advised not to use global variables as this can lead to a lot of mess in the software and problems with debugging. It seems you even never use it anywhere else, so why the global here?

I need a variable to remember pytest has been called for this case, so I use pytest_running_dir to identify this directory has been visited then pytest has been called. In C I could use "static" variable but in python I couldn't find an equivalence, so I use "global pytest_running_dir"

PerMac · 2021-03-05T12:17:27Z

scripts/pylib/twister/harness.py

+        self.running_dir = None
+        self.source_dir = None
+        self.pytest_root = 'pytest'
+        self.pytest_timeout = 15


these are not needed by default, so why initializing them here? Your class Pytest inherits from Harness. You can overcharge the __init__ method in Pytest by calling the parent __init__ and adding extra stuff you need. This will help in keeping the modularity and coherence of the code
https://appdividend.com/2019/01/22/python-super-function-example-super-method-tutorial/

updated in #33227, move to pytest.configure()

PerMac · 2021-03-05T12:32:42Z

scripts/pylib/twister/harness.py

+            self.running_dir = config.get('running_dir', None)
+            self.source_dir = config.get('source_dir', None)
+            self.pytest_root = config.get('pytest_root', 'pytest')
+            self.pytest_timeout = config.get('pytest_timeout', 15)


As above. You can overwrite the configure() method in Pytest class. You can see below, in Console(Harness) how this is done there.

updated in #33227, move to pytest.configure()

PerMac · 2021-03-05T12:34:45Z

scripts/pylib/twister/harness.py

+        '''
+        global pytest_running_dir
+        if self.running_dir is pytest_running_dir:
+            return


I have no idea what this logic suppose to do. Why would you return like that just at the beginning of handle()?

To make sure this handle be called only once. Every line output from console will cause this handle be called. I need a variable to remember pytest has been called for this case, so I use pytest_running_dir to identify this directory has been visited then pytest has been called. In C I could use "static" variable but in python I couldn't find an equivalence, so I use
"global pytest_running_dir"

PerMac · 2021-03-05T12:57:05Z

tests/pytest/pytest_sample/src/main.c

+/*
+ * Copyright (c) 2020 Intel Corporation
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+#include <ztest.h>
+void test_pytest(void)
+{
+	TC_PRINT("Hello world\n");
+}
+
+void test_main(void)
+{
+	ztest_test_suite(test_pytest,
+			 ztest_unit_test(test_pytest)
+			 );
+
+	ztest_run_test_suite(test_pytest);
+}


@nashif Were you able to follow the design of this "feature"? It builds and executes some hello_word sample, which is in fact not tested. It is just a "workaround" to fit the existing structure of a workflow, where something has to be built. Then the sample is executed and the output line is "handled". In fact, the line is not checked but serves as a trigger for calling pytest. The test for pytest has nothing to do with the building sample.

PerMac · 2021-03-05T15:11:51Z

scripts/pylib/twister/harness.py

+			'-q'
+        ]
+
+        with subprocess.Popen(cmd) as proc:


you should pipe the stdout and stderr so the messages from tests are not printed to the output when not needed (no verbose used)

updated in #33227

PerMac · 2021-03-08T08:44:51Z

scripts/pylib/twister/twisterlib.py

+        harness.running_dir = self.build_dir
+        harness.source_dir = self.sourcedir


Why you've added this to all already existing handlers, where it is not needed?

updated in #33227, move to pytest.configure()

PerMac · 2021-03-08T08:45:07Z

scripts/pylib/twister/twisterlib.py

+        harness.running_dir = self.build_dir
+        harness.source_dir = self.sourcedir


Why you've added this to all already existing handlers, where it is not needed?

updated in #33227, move to pytest.configure()

PerMac · 2021-03-08T08:45:13Z

scripts/pylib/twister/twisterlib.py

+        harness.running_dir = self.build_dir
+        harness.source_dir = self.sourcedir


Why you've added this to all already existing handlers, where it is not needed?

updated in #33227

PerMac · 2021-03-08T09:12:08Z

tests/pytest/pytest_sample/testcase.yaml

@@ -0,0 +1,4 @@
+tests:
+  pytest.sample:
+    platform_allow: qemu_x86 qemu_x86_64


This causes the same test to be run 2x. As I understand, the platform chosen has no impact on the actual test. It just serves as a stub so sth is build and handling lines can trigger calling pytest. I would say that there should be just a single "native_posix" if we accept this workflow.

updated in #33227

PerMac · 2021-03-08T09:41:31Z

tests/pytest/pytest_sample/pytest/test_ctf.py

+    if not os.path.exists(cmdopt):
+        assert 0


why not just:

assert os.path.exists(cmdopt)

?

Thank you, I've updated it in #33227

nashif · 2021-03-08T17:30:16Z

a

Comments added. In general, I find the workflow for pytest test convoluted and the implementation to be suboptimal.

agree on the implementation, but it seems the purpose of this is being missed here.

The idea behind adding this mode is to be able to use pytest in this case to do post run tests on build collateral and build output, for example, some subsystems generate binary data over UART (tracing) that could be evaluated after the test is run using pytest or we could be do post build analysis on generated binaries and linker scripts, something you can't do using current ztest.

Ideally we want to merge the results of pyests in the report as additional testcases, this is not being done here but we have a path to get this in after the restructuring of test results is complete.

PerMac · 2021-03-09T08:45:53Z

agree on the implementation, but it seems the purpose of this is being missed here.

@nashif I am not sure who you address here :) By convoluted I meant how the workflow is done in the PR:

some phony sample is built just to trigger output handler which calls for pytest. The sample built has nothing to do with pytest

nashif · 2021-03-09T12:11:42Z

* some phony sample is built just to trigger output handler which calls for pytest. The sample built has nothing to do with pytest

the sample can be anything if you for example checking the output and verifying that for example files are being produced in some format you expect. I would agree that the sample should show some real use case and not just call some useless asserts.

LeiW000 · 2021-03-16T02:51:40Z

@mengxianglinx, do you need to close this PR as #33227 has been filed.

The background on why Xianglin had to file #33227: @youhuazhu, who is the owner of this PR, does not work on the Zephyr project anymore. Xianglin is not able to update this PR with his account. So he has filed #33227.

nashif · 2021-04-19T01:56:33Z

obsoleted by #33227

youhuazhu requested a review from nashif as a code owner October 22, 2020 02:09

github-actions bot added the area: Sanitycheck Sanitycheck has been renamed to Twister label Oct 22, 2020

youhuazhu force-pushed the sanitycheck_pytest branch 2 times, most recently from aeaf374 to 607e95c Compare October 22, 2020 02:15

youhuazhu requested review from LeiW000, mengxianglinx and enjiamai October 22, 2020 02:15

LeiW000 approved these changes Oct 22, 2020

View reviewed changes

LeiW000 requested a review from galak October 22, 2020 05:46

youhuazhu force-pushed the sanitycheck_pytest branch from 607e95c to c46a374 Compare October 22, 2020 07:49

github-actions bot added the area: Documentation label Oct 22, 2020

nashif requested a review from PerMac October 22, 2020 12:02

nashif requested changes Oct 26, 2020

View reviewed changes

scripts/sanity_chk/harness.py Outdated Show resolved Hide resolved

doc/guides/test/sanitycheck.rst Outdated Show resolved Hide resolved

scripts/sanity_chk/harness.py Outdated Show resolved Hide resolved

youhuazhu force-pushed the sanitycheck_pytest branch from c46a374 to 1aad767 Compare October 27, 2020 07:56

github-actions bot added the area: Tests Issues related to a particular existing or missing test label Oct 27, 2020

youhuazhu force-pushed the sanitycheck_pytest branch 2 times, most recently from edd2ba3 to aa90791 Compare October 27, 2020 08:07

nashif reviewed Oct 27, 2020

View reviewed changes

scripts/sanity_chk/harness.py Outdated Show resolved Hide resolved

nashif reviewed Oct 27, 2020

View reviewed changes

scripts/sanity_chk/harness.py Outdated Show resolved Hide resolved

nashif reviewed Oct 27, 2020

View reviewed changes

scripts/sanity_chk/sanitylib.py Outdated Show resolved Hide resolved

nashif reviewed Oct 27, 2020

View reviewed changes

doc/guides/test/sanitycheck.rst Outdated Show resolved Hide resolved

nashif reviewed Oct 27, 2020

View reviewed changes

scripts/sanity_chk/harness.py Outdated Show resolved Hide resolved

youhuazhu force-pushed the sanitycheck_pytest branch from aa90791 to 6635648 Compare October 28, 2020 02:17

youhuazhu requested a review from nashif October 28, 2020 07:23

PerMac requested changes Dec 11, 2020

View reviewed changes

youhuazhu force-pushed the sanitycheck_pytest branch 6 times, most recently from 59589c6 to 12751d9 Compare December 14, 2020 13:07

PerMac requested changes Dec 23, 2020

View reviewed changes

zephyrbot added the area: Twister Twister label Jan 8, 2021

zephyrbot requested a review from nashif January 8, 2021 13:28

zephyrbot assigned nashif Jan 8, 2021

nashif removed the area: Sanitycheck Sanitycheck has been renamed to Twister label Jan 11, 2021

youhuazhu force-pushed the sanitycheck_pytest branch 3 times, most recently from ebb7c38 to ff2c862 Compare January 15, 2021 07:53

youhuazhu added 2 commits January 15, 2021 15:58

tests: pytest: add an example for pytest

f0af9b4

In this example, python test case get the running directory by handling the "--cmdopt" option passed by pytest. Signed-off-by: YouhuaX Zhu <youhuax.zhu@intel.com>

youhuazhu force-pushed the sanitycheck_pytest branch 2 times, most recently from 762ecb5 to f0af9b4 Compare January 15, 2021 08:09

nashif requested changes Mar 6, 2021

View reviewed changes

PerMac requested changes Mar 8, 2021

View reviewed changes

mengxianglinx mentioned this pull request Mar 11, 2021

Twister pytest #33227

Merged

nashif closed this Apr 19, 2021

+                  ::
+                      tests:
+                        test:

		harness.running_dir = self.build_dir
		harness.source_dir = self.sourcedir

sanitycheck: Add a feature which can handle pytest script. #29427

sanitycheck: Add a feature which can handle pytest script. #29427

Conversation

youhuazhu commented Oct 22, 2020 • edited by mengxianglinx

galak commented Oct 22, 2020

youhuazhu commented Oct 22, 2020

LeiW000 commented Oct 22, 2020

nashif commented Oct 22, 2020

youhuazhu commented Oct 29, 2020

LeiW000 commented Nov 5, 2020

Choose a reason for hiding this comment

mengxianglinx Mar 23, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PerMac left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LeiW000 commented Mar 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PerMac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nashif commented Mar 8, 2021

PerMac commented Mar 9, 2021

nashif commented Mar 9, 2021

LeiW000 commented Mar 16, 2021

nashif commented Apr 19, 2021

youhuazhu commented Oct 22, 2020 •

edited by mengxianglinx

mengxianglinx Mar 23, 2021 •

edited

PerMac left a comment •

edited