Use faulthandler instead of isys signal handlers

[VladimirSlavik]

This significantly changes outputs, but provides eventually the same kinds of information, only in different order.

Previously:

• Print that a signal was received, and which one.
• Log to syslog that this happened.
• Print userland stack trace of main process.
• Save core dump:
  • always to /tmp/anaconda.core.
  • always, independent of system settings
  • somehow very slowly - maybe forking python and running gcore isn't the most performant idea
  • file size ca. 1 GB
• Journal gets only the syslog message.

Now:

• Print that a fatal error happened, no mention of signals.
• No unique message in syslog and journal to identify that "anaconda" crashed.
• Print python stack of all the threads in the main process.
• Save core dump:
  • to default location
  • according to system settings
  • needs invoking coredumpctl manually to work with the actual core dump
  • blazing fast saving compared to previous state
  • size ca. 430 MB, initally stored compressed to ca. 60 MB
  • apparently requires more debuginfo downloads, system can run out of space or memory
• Print userland stack traces of all threads to journal.

In both cases, using the (coredumpctl+)gdb+debuginfod+dnf toolchain leads to a successful interactive debugging session.

[VladimirSlavik]

/kickstart-test --testtype smoke

[rvykydal]

Looks good to me, thank you!
I wonder if we need to add / update strings in log monitor of kickstart tests. May be worth checking, but not blocking the PR.

[VladimirSlavik]

Certainly, and livemedia-creator too, and who knows how many other places.

[VladimirSlavik]

A few more details:

• Looks like we might have to watch for a line like "Process [0-9]+ (anaconda) of user [0-9]+ dumped core" - not sure where: kickstart tests, lorax? Fortunately the logging "filter" is mostly shared.
• The signal is mentioned in coredumpctl listing (by name instead of number).
• The "good enough" userland stack traces (same as before) are visible with coredumpctl info (...) even without going into gdb and loading all the debuginfos.
• coredumpctl info without query gives the details of the last recorded core dump, which should be enough in most cases.

I will amend the commit message with that.

[VladimirSlavik]

Based on the above, I think we can merge this and not lose anything.

[VladimirSlavik]

/kickstart-test --testtype smoke

[poncovka]

Thank you!

[VladimirSlavik]

Looks like we might have to watch for a line like "Process [0-9]+ (anaconda) of user [0-9]+ dumped core" - not sure where: kickstart tests, lorax?

@bcl just a heads up that this might change how fatal errors look...

[bcl]

The code that lmc uses is here - https://github.com/weldr/lorax/blob/f33-branch/src/pylorax/monitor.py#L38

So as long as the new output has either 'Traceback' or 'Call Trace:' in it lmc will catch it.

[VladimirSlavik]

Unfortunately that's none of that.

• In shell: Fatal Python error: (...)
• In syslog: CRIT systemd-coredumpctl:Process [0-9]+ (anaconda) of user [0-9]+ dumped core.
• In journal: systemd-coredumpctl\[[0-9]+\]: \[■\] Process [0-9]+ \(anaconda\) of user [0-9]+ dumped core.

I'll make a PR.