coredumpd: improve backtraces by downloading debuginfo on the fly via elfutils/debuginfod

[fche]

Is your feature request related to a problem? Please describe.
When computing a traceback, systemd-coredump can produce higher quality results if debuginfo is available. Current elfutils has some automatic downloading capability, but this is made difficult to configure/use from systemd-coredump@ due to the numerous security constraints.

Describe the solution you'd like
Consider allowing a coredump@ service mode that relaxes the security constraints or data flow just enough to launch the elfutils debuginfod (https://sourceware.org/elfutils/Debuginfod.html) client code for the binaries/coredumps being processed. To trigger the attempt, it is enough to set a $DEBUGINFOD_URLS environment variable for the service; elfutils does the rest. But since this is by necessity making contact to a network service that involves downloading, so IPAddressDeny, RestrictAccessFamilies, SystemCallFilter, and probably other settings need to be relaxed. There may be a way of factoring out this to a less locked-down helper process or service while leaving the main one locked up tight. For example, the /usr/bin/eu-stack program can do similar stack backtracing as the code in your src/coredump/stacktrace.c.

If this capability is of any interest, we may be able to help prototype it under your guidance.

[poettering]

hmm, so losening the sandbox for this is less than ideal, i mean parsing coredumps is not without risk I'd say, it's a parser par excellence, and hence should benefit greatly from the sandboxing.

I think it would definitely make sense to make this on opt-in thing. maybe one approach might be to simply try to contact the servers and if that doesn't work, quickly give up gracefully. Thus, in our usual sandbox this wouldn't work, but if you drop-in /etc/systemd/system/systemd-coredump@.service.d/unsandbox.conf or so which turns off the network sandboxing stuff we would magically start using this and generate more elaborate backtraces.

I mean, this is what it comes down to (at least on fedora), right? whether you get full blown backtraces or just the minimal ones via minidebuginfo, right?

[fche]

so losening the sandbox for this is less than ideal

Understood. Perhaps: have a -second- service that monitors the journal for effluents of the first, and fills in a new related record, based on network data?

whether you get full blown backtraces or just the minimal ones via minidebuginfo, right?

I believe so - you get accurate backtraces in light of inlining etc., rather than heuristic.

[owtaylor]

Ran into a similar situation with Flatpak where to debug a process you have to loosen the sandbox - which could change the behavior you are trying to debug. Though it wouldn't be trivial to get the life-cycle management right, I'm wondering about a mode where elfutils can talk to a debuginfo server on a unix domain socket, and that proxies to DEBUGINFOD_URLS (and handles caching?) See flatpak/flatpak#4222

[fche]

Or arrange to run a socat type process to bridge traffic between localhost:PORT and a unix-domain socket. (It would be nice if the CURL system allowed a URL naming pattern for http-over-unix-domain-socket but I don't see one.)

[owtaylor]

Requiring a process inside the sandbox to bridge the traffic would make the setup even more complex. It looks like libcurl allows connecting to a unix socket (https://curl.se/libcurl/c/CURLOPT_UNIX_SOCKET_PATH.html) but not via URL, so elfutils would have to handle extra syntax in some form.

[fche]

Another option for that scenario could be curl's native support for proxies: $http_proxy or $CURLOPT_PROXY (socks5). I'm not super keen on adding native http-over-unixfd to libdebuginfod proper.