Look for problems in the system

[mnowaksuse]

POO#11930

This new sub problem_detection() detects problems in the installed
system and uploads log to worker. Currently it looks into systemd
journal, coredumpctl, systemctl et all for core dumps, signs of python
traceback, general problems in the journal etc. (Other proposals
welcome.)

I implemented POO#11930 not as an enhancement to textinfo.pm test but as
a function which might be called from different places (installed system
and even installer itself), and multiple times. Atm I execute it from
console/consoletest_setup.pm and x11/shutdown.pm. Problem of current
approach is that it adds 5-6 minutes to the test suite per execution.

Verification run: http://assam.suse.cz/tests/2678#downloads.

[okurz]

Interesting approach. I like the idea in general. What I am not so sure about is that you are calling this in "consoletest_setup" which is a part of a lot of scenarios and will further slow down the execution in cases when everything is just fine. Also, as complexity increases in your problem analysis, jobs can also fail in there. I suggest to better separate what is "setup", what is "gather generic information" and "do analysis of a real problem".

For a start, maybe you can even in the current state move out of the logs gathering from "consoletest_setup" and make that module just what it says, a "setup", e.g. only including setting up the serial port, consoles, making sure packagekit is stopped, etc.

The sub problem_detection by itself looks good but can we start by calling it in post_fail_hook for a start and discuss later where it should be called even in case of passing test modules?

[mnowaksuse]

http://assam.suse.cz/tests/2774#step/consoletest_setup/31

Removed some duplicities from console/consoletest_setup, moved analysis to export_logs(). problem_detection() is now being called from export_logs() only but can be called from tests as well where needed. problem_detection()'s analysis is now bit faster as it's result is packed and uploaded only once.

Still:

1. Calling save_and_upload_log() feels overcomplicated.
2. My feeling is that once core dump is found on the system, we reviewer should be notified as it's probably something we want to be fixed anyway. But using assert_script_run() just end the sub (as it should): http://assam.suse.cz/tests/2777#step/consoletest_setup/57.

[okurz]

perl beginner question: As $args should always exist as a variable, does it work to just write

script_run("$cmd | tee $file", $args->{timeout});

in any case without getting a warning?

[mnowaksuse]

It actually works as use warnings is not present. Thanks for the hint.

[okurz]

can't access your machine, is it down?

[mnowaksuse]

can't access your machine, is it down?

It's back online now.

[mnowaksuse]

New verify run: http://assam.suse.cz/tests/2785#step/consoletest_setup/32.

[nilxam]

we also need it in each succeeded run, please do not remove it or only uploading when the test failed.

[okurz]

LGTM

wanna remove the WIP label?

[nilxam]

LGTM