Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][DISCUSSION] Fully replace QA_TESTSUITE approach by QA_TESTSET #6955

Closed
wants to merge 1 commit into from

Conversation

okurz
Copy link
Member

@okurz okurz commented Mar 4, 2019

#4132 with title
"qa_automation: add new design for QA:HEAD tests" added the use of the test
variable QA_TESTSUITE whereas qam uses QA_TESTSET only. Checked with
https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module
and found no QAM job running execute_test_run.
https://openqa.suse.de/admin/test_suites shows that there are mainly QAM jobs
using QA_TESTSET however some others as well. QA_TESTSUITE is not used for
anything with a prefix like "qam". To me it looks like both are doing more or
less the equivalent except that based on QA_TESTSET shows the output of
individual test modules and also there is no module "execute_test_run" that
would fail and is hard to label.

The approach based on QA_TESTSET seems to provide much better visualization of
test results using the more recent openQA feature to show "External results".

Related progress issue: https://progress.opensuse.org/issues/44138

os-autoinst#4132 with title
"qa_automation: add new design for QA:HEAD tests" added the use of the test
variable QA_TESTSUITE whereas qam uses QA_TESTSET only. Checked with
https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module
and found no QAM job running execute_test_run.
https://openqa.suse.de/admin/test_suites shows that there are mainly QAM jobs
using QA_TESTSET however some others as well. QA_TESTSUITE is not used for
anything with a prefix like "qam". To me it looks like both are doing more or
less the equivalent except that based on QA_TESTSET shows the output of
individual test modules and also there is no module "execute_test_run" that
would fail and is hard to label.

The approach based on QA_TESTSET seems to provide much better visualization of
test results using the more recent openQA feature to show "External results".

Related progress issue: https://progress.opensuse.org/issues/44138
@okurz okurz added the notready label Mar 4, 2019
@okurz
Copy link
Member Author

okurz commented Mar 4, 2019

@cachen In https://progress.opensuse.org/issues/44138 I am trying to harmonize the workflow of QA_TESTSUITE and QA_TESTSET. As far as I understand the approach based on QA_TESTSET provides the equivalent except that it has better visualization in openQA. You added the approach based on QA_TESTSUITE in #4132 so I would like to ask you if you think that this PR makes sense and we should go with QA_TESTSET or did I miss something?

@cachen
Copy link
Contributor

cachen commented Apr 23, 2019

@okurz sorry for the late reply! #4132 was fixing test workflow unreadable issue, and aims to reduce some unnecessary dependence packages/workflows(such as qa_testset_automation), it wasn't for fixing log output issue. I would suggest to use QA_TESTSUITE method.
Since now openQA has External results function works well with the converted junit xml, for better output the test logs to the External results, the way could be move below code from qa_run.pm to execute_test_run.pm to replace the log upload and junit converting functions. Then we will able to keep all purposes.

# Upload test logs
my $tarball = "/tmp/qaset.tar.bz2";
assert_script_run("tar cjf '$tarball' -C '/var/log/' 'qaset'");
upload_logs($tarball, timeout => 600);
my $log = $self->system_status();
upload_logs($log, timeout => 100);

# JUnit xml report
assert_script_run("/usr/share/qa/qaset/bin/junit_xml_gen.py -n 'regression' -d -o /tmp/junit.xml /var/log/qaset");

parse_junit_log("/tmp/junit.xml");

@foursixnine foursixnine mentioned this pull request May 22, 2019
@okurz
Copy link
Member Author

okurz commented Aug 29, 2019

Apparently there is not enough interest to gain efficiency by this approach also as @Soulofdestiny closed https://progress.opensuse.org/issues/49118 so I will close this PR as well

@okurz okurz closed this Aug 29, 2019
@okurz okurz deleted the enhance/delete_qa_testsuite branch August 29, 2019 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants