Upload crash dump when crash utility fails

[jknphy]

Upload crash dump when crash utility fails. As requested in https://bugzilla.suse.com/show_bug.cgi?id=1090659 we need some way to provide the dump to be analyzed.

• Related ticket: https://progress.opensuse.org/issues/33376
• Verification run: for ppc we cannot reach that point, our shared worker does not run properly kdump, but similar code (excuding the if/else condition) that I did for 64bit looks good: http://dhcp254.suse.cz/tests/1184#downloads. The content of those files could reveal why is failing performing virtual-to-physical translation.

[rwx788]

Don't we need to increase timeout here too?

[rwx788]

I guess we can skip this run as we already executed it on L51

[okurz]

why record the soft failure and save the tar file but not upload it on non-sle-15+?

[okurz]

Already in the original code the $crash_cmd was called twice? Would it work to call script_run "$crash_cmd |& tee /dev/$serialdev and then check with wait_serial? I guess not in this order as the serial output is consumed by script_run checking for the exit status. But probably script_run '…', 0 and an explicit wait_serial for both the output and the exit code would work if you want to play around with that, e.g. something like

script_run("$crash_cmd |& tee /dev/$serialdev; echo crash-status-\$? > /dev/$serialdev", 0);
if (!wait_serial('PANIC', 600)) {
    diag "crash could not find crash file";
    …
    return;
}
wait_serial('crash-status-0') or die "crash did not complete successfully";

[jknphy]

The only way that I found to keep the timeout, evaluate both possible errors and upload the file is in the latest code. Basically this is what it does:

• Run the script and immediately start to read the serial output.
• Search for "PANIC: ", which in my opinion was incomplete so I extended, because some text need to appear after "PANIC:" reflecting how to dump was triggered.
• Search for the output of the command.
• Search both at the same time with one regex, I found it more reliable than searching sequentially
• Avoid to wait for the whole timeout using if instead of unless
• Upload the file (reminder: main goal of this task).

VR: http://dhcp254.suse.cz/tests/1212#step/kdump_and_crash/52

[okurz]

Hm, looks a bit generic because now this branch would always be hit in case the above wait_serial either did not find PANIC, non-zero exit code or timeout.
You could check the actual return of wait_serial in detail

[jknphy]

I realized that in ppc it fails but the command ended, so we can do it in post_fail_hook in a cleaner way. We just need to remove the post_fail_hook when the bug is solved. The code was starting to be very cumbersome using the serial output and in this way we still check if the crash is ok and if the crash have a PANIC written. The post_fail_hook is generic for any kind of problem about crash, so it will be useful for future.
VR: http://dhcp254.suse.cz/tests/1222#step/kdump_and_crash/65
VR forcing the failing to see how it looks (searching, for instance XPANIC, instead ): http://dhcp254.suse.cz/tests/1223#step/kdump_and_crash/65

[okurz]

Nice, much cleaner :)