[PROPOSAL]Schedule systemd upstream tests dynamically #15083

mloviska · 2022-06-15T08:16:47Z

Tests provided by systemd upstream are packaged in systemd-testsuite.
Each test case is defined in folder with naming convention
TEST-\d{2}-$testname. The runner schedules all test cases that are
present in the provided package. Tests are preferably running in nswapn,
however some of them require qemu.
Users can override the default schedule either excluding certain test
cases provided by a regex. For debug purposes users can provide a list
of tests that will override the default schedule.

github-actions · 2022-06-15T08:17:15Z

Great PR! Please pay attention to the following items before merging:

Files matching lib/**.pm:

Consider adding or extending unit tests in t/

This is an automatically generated QA checklist based on modified files

mloviska · 2022-06-15T08:23:52Z

~~This PR is only a proposal, as I haven't seen how the test are schedule in the current state.~~

mloviska · 2022-07-04T12:15:25Z

VRs:

tests/systemd_testsuite/runner.pm

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm

tests/systemd_testsuite/runner.pm

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm

tests/systemd_testsuite/runner.pm

mdoucha

LGTM

lib/main_common.pm

products/opensuse/main.pm

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm

b10n1k · 2022-07-07T20:38:41Z

tests/systemd_testsuite/runner.pm

@@ -0,0 +1,70 @@
+# SUSE's openQA tests
+#
+# Copyright 2019-2020 SUSE LLC


i think you can neglect the dates. Otherwise i think it needs update

b10n1k

Looks nice @mloviska. just minor comments.

mloviska · 2022-07-08T10:24:52Z

Hello @tblume, may I ask you for review as well? This way we can do not need to a file per test case in tests folder, and it should be easier to exclude several test cases.

tblume · 2022-07-08T10:42:30Z

Hello @tblume, may I ask you for review as well? This way we can do not need to a file per test case in tests folder, and it should be easier to exclude several test cases.

I'm happy that the systemd-testsuite code gets some maintenance from you guys!
You know much better than I do.
I agree to the changes and would like to test them on my local openqa machine.
What would be the appropriate openqa-clone-job command to test it?

mloviska · 2022-07-08T11:22:03Z

What would be the appropriate openqa-clone-job command to test it?

Should be fairly easy, I have got some examples cloning opensuse-Tumbleweed-DVD-x86_64-Build20220707-extra_tests_textmode@64bit.

Full schedule ( all test cases included in the package )
/usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= --skip-chained-deps
Exclude TEST-06-SELINUX only
/usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= SYSTEMD_EXCLUDE=TEST-06-SELINUX --skip-chained-deps
Run only TEST-0.*
/usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= SYSTEMD_EXCLUDE="TEST-[1-9].*" --skip-chained-deps
Include only TEST-01-BASIC and change default test case parameters to UNIFIED_CGROUP_HIERARCHY="no" and TEST_PREFER_NSPAWN=0
/usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= SYSTEMD_INCLUDE=TEST-01-BASIC SYSTEMD_UNIFIED_CGROUP="no" SYSTEMD_NSPAWN=0 --skip-chained-deps

tblume · 2022-07-08T13:53:22Z

What would be the appropriate openqa-clone-job command to test it?

Should be fairly easy, I have got some examples cloning opensuse-Tumbleweed-DVD-x86_64-Build20220707-extra_tests_textmode@64bit.
1. [Full schedule ( all test cases included in the package )](http://kepler.suse.cz/tests/17649#)
   /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= --skip-chained-deps

Hm, I get a strange error when executing the tests therewith (booting from a previously created image):

--<
Test died: Failed to wait for login prompt at sle/lib/serial_terminal.pm line 134.
serial_terminal::login("root", "# ") called at sle/lib/susedistribution.pm line 816
susedistribution::activate_console(Distribution::Sle::15_current=HASH(0x556480ffe8e8), "root-virtio-terminal") called at /usr/lib/os-autoinst/testapi.pm line 1769
testapi::select_console("root-virtio-terminal") called at sle/lib/opensusebasetest.pm line 1320
opensusebasetest::select_serial_terminal(runner=HASH(0x5564812f9de0)) called at sle/tests/systemd_testsuite/runner.pm line 35
runner::run(runner=HASH(0x5564812f9de0), OpenQA::Test::RunArgs=HASH(0x556481bee918)) called at /usr/lib/os-autoinst/basetest.pm line 324
eval {...} called at /usr/lib/os-autoinst/basetest.pm line 322
basetest::runtest(runner=HASH(0x5564812f9de0)) called at /usr/lib/os-autoinst/autotest.pm line 360
eval {...} called at /usr/lib/os-autoinst/autotest.pm line 360
autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 236
eval {...} called at /usr/lib/os-autoinst/autotest.pm line 236
autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 287
autotest::ANON(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090), CODE(0x556482ae5990)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 488
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090)) called at /usr/lib/os-autoinst/autotest.pm line 289
autotest::start_process() called at /usr/bin/isotovideo line 256
--<

you can take a look at my testmachine at: http://knopfler.arch.suse.de/tests/8#

mloviska · 2022-07-08T14:09:39Z

What would be the appropriate openqa-clone-job command to test it?

Should be fairly easy, I have got some examples cloning opensuse-Tumbleweed-DVD-x86_64-Build20220707-extra_tests_textmode@64bit.
1. [Full schedule ( all test cases included in the package )](http://kepler.suse.cz/tests/17649#)
   /usr/share/openqa/script/clone_job.pl --from https://openqa.opensuse.org --host localhost 2458023 TEST=systemd SYSTEMD_TESTSUITE=1 YAML_SCHEDULE= --skip-chained-deps
Hm, I get a strange error when executing the tests therewith (booting from a previously created image):

--< Test died: Failed to wait for login prompt at sle/lib/serial_terminal.pm line 134. serial_terminal::login("root", "# ") called at sle/lib/susedistribution.pm line 816 susedistribution::activate_console(Distribution::Sle::15_current=HASH(0x556480ffe8e8), "root-virtio-terminal") called at /usr/lib/os-autoinst/testapi.pm line 1769 testapi::select_console("root-virtio-terminal") called at sle/lib/opensusebasetest.pm line 1320 opensusebasetest::select_serial_terminal(runner=HASH(0x5564812f9de0)) called at sle/tests/systemd_testsuite/runner.pm line 35 runner::run(runner=HASH(0x5564812f9de0), OpenQA::Test::RunArgs=HASH(0x556481bee918)) called at /usr/lib/os-autoinst/basetest.pm line 324 eval {...} called at /usr/lib/os-autoinst/basetest.pm line 322 basetest::runtest(runner=HASH(0x5564812f9de0)) called at /usr/lib/os-autoinst/autotest.pm line 360 eval {...} called at /usr/lib/os-autoinst/autotest.pm line 360 autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 236 eval {...} called at /usr/lib/os-autoinst/autotest.pm line 236 autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 287 autotest::ANON(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326 eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326 Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090), CODE(0x556482ae5990)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 488 Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x556482b51090)) called at /usr/lib/os-autoinst/autotest.pm line 289 autotest::start_process() called at /usr/bin/isotovideo line 256 --<

you can take a look at my testmachine at: http://knopfler.arch.suse.de/tests/8#

I assume it happens because the VM has been shutdown. I am always using jobs that are chained after the installation as extra_tests_dracut. Installation jobs, mainly the ones that create a HDD set openQA variable INSTALLONLY=1 that should prevent other post installation tests to be scheduled.

http://kepler.suse.cz/tests/17660#

tests/systemd_testsuite/runner.pm

cfconrad

LGTM!
And thx for showing autotest::loadtest() during runtime!!

Tests provided by systemd upstream are packaged in *systemd-testsuite*. Each test case is defined in folder with naming convention `TEST-\d{2}-$testname`. The runner schedules all test cases that are present in the provided package. Tests are preferably running in nswapn, however some of them require qemu. Users can override the default schedule either excluding certain test cases provided by a regex. For debug purposes users can provide a list of tests that will override the default schedule.

mloviska · 2022-07-26T11:27:58Z

Hello all, can we merge it if there are no objections?

mloviska changed the title ~~Schedule systemd upstream tests dynamically~~ [PROPOSAL]Schedule systemd upstream tests dynamically Jun 15, 2022

mloviska force-pushed the systemd_kickoff branch from aa7e977 to 5df8238 Compare June 15, 2022 08:26

mloviska force-pushed the systemd_kickoff branch from 5df8238 to b5d31a8 Compare June 23, 2022 16:03

mloviska requested review from jlausuch, cfconrad and foursixnine July 4, 2022 14:05

mloviska marked this pull request as ready for review July 4, 2022 14:06

foursixnine reviewed Jul 4, 2022

View reviewed changes

tests/systemd_testsuite/runner.pm Show resolved Hide resolved

foursixnine reviewed Jul 4, 2022

View reviewed changes

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm Outdated Show resolved Hide resolved

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm Outdated Show resolved Hide resolved

tests/systemd_testsuite/runner.pm Outdated Show resolved Hide resolved

mloviska force-pushed the systemd_kickoff branch from b5d31a8 to b1993be Compare July 5, 2022 11:57

mdoucha requested changes Jul 7, 2022

View reviewed changes

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm Outdated Show resolved Hide resolved

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm Show resolved Hide resolved

tests/systemd_testsuite/runner.pm Outdated Show resolved Hide resolved

mloviska force-pushed the systemd_kickoff branch from b1993be to a56e11d Compare July 7, 2022 14:38

mdoucha approved these changes Jul 7, 2022

View reviewed changes

b10n1k reviewed Jul 7, 2022

View reviewed changes

lib/main_common.pm Show resolved Hide resolved

b10n1k reviewed Jul 7, 2022

View reviewed changes

lib/main_common.pm Show resolved Hide resolved

b10n1k reviewed Jul 7, 2022

View reviewed changes

products/opensuse/main.pm Show resolved Hide resolved

b10n1k reviewed Jul 7, 2022

View reviewed changes

tests/systemd_testsuite/prepare_systemd_and_testsuite.pm Show resolved Hide resolved

b10n1k reviewed Jul 7, 2022

View reviewed changes

foursixnine approved these changes Jul 8, 2022

View reviewed changes

mloviska force-pushed the systemd_kickoff branch from a56e11d to 04744e7 Compare July 8, 2022 13:03

cfconrad reviewed Jul 14, 2022

View reviewed changes

tests/systemd_testsuite/runner.pm Outdated Show resolved Hide resolved

cfconrad reviewed Jul 14, 2022

View reviewed changes

tests/systemd_testsuite/runner.pm Outdated Show resolved Hide resolved

cfconrad approved these changes Jul 14, 2022

View reviewed changes

mloviska added 2 commits July 14, 2022 17:43

Schedule systemd upstream tests

aad6df6

mloviska force-pushed the systemd_kickoff branch from 04744e7 to aad6df6 Compare July 14, 2022 15:43

mloviska merged commit 9522ec0 into os-autoinst:master Aug 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PROPOSAL]Schedule systemd upstream tests dynamically #15083

[PROPOSAL]Schedule systemd upstream tests dynamically #15083

mloviska commented Jun 15, 2022

github-actions bot commented Jun 15, 2022

mloviska commented Jun 15, 2022 •

edited

mloviska commented Jul 4, 2022

mdoucha left a comment

b10n1k Jul 7, 2022

b10n1k left a comment

mloviska commented Jul 8, 2022

tblume commented Jul 8, 2022

mloviska commented Jul 8, 2022

tblume commented Jul 8, 2022

mloviska commented Jul 8, 2022

cfconrad left a comment

mloviska commented Jul 26, 2022

[PROPOSAL]Schedule systemd upstream tests dynamically #15083

[PROPOSAL]Schedule systemd upstream tests dynamically #15083

Conversation

mloviska commented Jun 15, 2022

github-actions bot commented Jun 15, 2022

mloviska commented Jun 15, 2022 • edited

mloviska commented Jul 4, 2022

mdoucha left a comment

Choose a reason for hiding this comment

b10n1k Jul 7, 2022

Choose a reason for hiding this comment

b10n1k left a comment

Choose a reason for hiding this comment

mloviska commented Jul 8, 2022

tblume commented Jul 8, 2022

mloviska commented Jul 8, 2022

tblume commented Jul 8, 2022

mloviska commented Jul 8, 2022

cfconrad left a comment

Choose a reason for hiding this comment

mloviska commented Jul 26, 2022

mloviska commented Jun 15, 2022 •

edited