Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredumpd: improve backtraces by downloading debuginfo on the fly via elfutils/debuginfod #14711

Open
fche opened this issue Jan 30, 2020 · 6 comments
Labels
coredump RFE 🎁

Comments

@fche
Copy link

@fche fche commented Jan 30, 2020

Is your feature request related to a problem? Please describe.
When computing a traceback, systemd-coredump can produce higher quality results if debuginfo is available. Current elfutils has some automatic downloading capability, but this is made difficult to configure/use from systemd-coredump@ due to the numerous security constraints.

Describe the solution you'd like
Consider allowing a coredump@ service mode that relaxes the security constraints or data flow just enough to launch the elfutils debuginfod (https://sourceware.org/elfutils/Debuginfod.html) client code for the binaries/coredumps being processed. To trigger the attempt, it is enough to set a $DEBUGINFOD_URLS environment variable for the service; elfutils does the rest. But since this is by necessity making contact to a network service that involves downloading, so IPAddressDeny, RestrictAccessFamilies, SystemCallFilter, and probably other settings need to be relaxed. There may be a way of factoring out this to a less locked-down helper process or service while leaving the main one locked up tight. For example, the /usr/bin/eu-stack program can do similar stack backtracing as the code in your src/coredump/stacktrace.c.

If this capability is of any interest, we may be able to help prototype it under your guidance.

@poettering
Copy link
Member

@poettering poettering commented Mar 30, 2021

hmm, so losening the sandbox for this is less than ideal, i mean parsing coredumps is not without risk I'd say, it's a parser par excellence, and hence should benefit greatly from the sandboxing.

I think it would definitely make sense to make this on opt-in thing. maybe one approach might be to simply try to contact the servers and if that doesn't work, quickly give up gracefully. Thus, in our usual sandbox this wouldn't work, but if you drop-in /etc/systemd/system/systemd-coredump@.service.d/unsandbox.conf or so which turns off the network sandboxing stuff we would magically start using this and generate more elaborate backtraces.

I mean, this is what it comes down to (at least on fedora), right? whether you get full blown backtraces or just the minimal ones via minidebuginfo, right?

@fche
Copy link
Author

@fche fche commented Mar 30, 2021

so losening the sandbox for this is less than ideal

Understood. Perhaps: have a -second- service that monitors the journal for effluents of the first, and fills in a new related record, based on network data?

whether you get full blown backtraces or just the minimal ones via minidebuginfo, right?

I believe so - you get accurate backtraces in light of inlining etc., rather than heuristic.

@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Apr 14, 2021

Ran into a similar situation with Flatpak where to debug a process you have to loosen the sandbox - which could change the behavior you are trying to debug. Though it wouldn't be trivial to get the life-cycle management right, I'm wondering about a mode where elfutils can talk to a debuginfo server on a unix domain socket, and that proxies to DEBUGINFOD_URLS (and handles caching?) See flatpak/flatpak#4222

@fche
Copy link
Author

@fche fche commented Apr 14, 2021

Or arrange to run a socat type process to bridge traffic between localhost:PORT and a unix-domain socket. (It would be nice if the CURL system allowed a URL naming pattern for http-over-unix-domain-socket but I don't see one.)

@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Apr 14, 2021

Requiring a process inside the sandbox to bridge the traffic would make the setup even more complex. It looks like libcurl allows connecting to a unix socket (https://curl.se/libcurl/c/CURLOPT_UNIX_SOCKET_PATH.html) but not via URL, so elfutils would have to handle extra syntax in some form.

@fche
Copy link
Author

@fche fche commented Apr 14, 2021

Another option for that scenario could be curl's native support for proxies: $http_proxy or $CURLOPT_PROXY (socks5). I'm not super keen on adding native http-over-unixfd to libdebuginfod proper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
coredump RFE 🎁
Development

No branches or pull requests

4 participants