Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: generate external DWARF debuginfo archives directly #51692

Open
ringerc opened this issue Mar 16, 2022 · 8 comments
Open

cmd/link: generate external DWARF debuginfo archives directly #51692

ringerc opened this issue Mar 16, 2022 · 8 comments

Comments

@ringerc
Copy link

@ringerc ringerc commented Mar 16, 2022

TL;DR: Add an -external-debuginfo command line argument to go tool link that causes the link command to write a .debug archive containing symbol tables and DWARF debuginfo for the executable being linked.

-w and/or -s will continue to have their current effects on the output executable. These flags will have no effect on the archive produced by -external-debuginfo, which will always include the full symbol table and DWARF debuginfo.

This could be implemented by using the functionality of elfutils eu-strip in cmd/link / pkg/tool/link via libelf.

For Windows targets a .pdb file could be produced, or external debuginfo support for Windows targets could be initially omitted if it's too hard to do at the same time.

Rationale

Kubernetes project binaries and container images are currently built and distributed with the -w -s linker flags, so DWARF debuginfo is omitted then the executable is stripped of all symbols.

This saves disk space by producing smaller binaries and container images, though it has no memory consumption or performance benefit at executable runtime.

The cost is that debugging such executables is extremely difficult - only numeric addresses are available, with no symbols for functions, variables, etc. If a problem only occurs in a production or near-production system and you want to inspect the running component, you will see backtraces like:

(gdb) bt
#0  0x000000000046fc63 in ?? ()
#1  0x0000000000432fc6 in ?? ()
#2  0x00000000048e2110 in ?? ()
#3  0x0000000000000080 in ?? ()
#4  0x0000000000000000 in ?? ()

In traditional gcc or llvm based C/C++ build tool chains a balance between executable size and debuggability is maintained using external debuginfo archives. These are supported by gcc, lldb and anything using elfutils or libgdb for symbol loading. Delve supports loading of external symbols from /usr/lib/debug/.build_id and /usr/lib/debug/{path} etc, like gdb and other tools.

Neither GNU ld and LLVM ld.lld provide a command line argument for writing external debuginfo archives. Instead the release process for most binaries includes a separate pass to split the original executable with debuginfo into separate stripped executable and debuginfo archive files using objcopy --only-keep-debug, eu-strip and objcopy --add-gnu-debuglink.

The golang toolchain tries to be an integrated single-stop shop without external linker dependencies etc, so golang projects including the official k8s distribution don't tend to use this workflow. They're release-size conscious so they just don't provide any debuginfo. They just build stripped binaries for release. This is convenient but makes it much more difficult to diagnose problems.

go/link should instead support directly generating an external debuginfo archive ready for upload to a debuginfod symbol server, inclusion in a container image's /usr/lib/debug and /usr/lib/debug/.build_id trees, and/or distribution in release tarballs.

Note that delve already supports debuginfod. If go/link provided an easy way to generate external symbol archives, the k8s project or an interested 3rd party could host a public debuginfod with debuginfo for all future release binaries for use in gdb, delve, lldb, etc.

For example in gdb, one might add https://debuginfod.k8s.io/ to the DEBUGINFOD_URLS env-var, add set debuginfod enabled to .gdbinit, and have symbols automatically downloaded on demand when debugging k8s release binaries. Even over a remote gdbserver. A similar approach is possible with Delve.

Symbols can also be downloaded on-demand from a debuginfod with eu-unstrip then used as normal local detached symbol archives, and eu-make-debug-archive can be used to make a full archive of a system or container's libraries and debuginfo for remote debugging use. But the symbols must be available for that to be possible.

Interim workaround

Golang binaries can be built with debuginfo and symbols (neither -s nor -w LDFLAGS should be used) then split into separate debuginfo with eu-strip for linux targets.

This requires each build script to implement features that require dependencies that aren't part of the main go toolchain. So it's unlikely to see wide adoption until and unless the golang toolchain itself provides a standard way to do it like it has with -s for stripped executables.

But it's not overly complicated:

mv mybin mybin.tmp
eu-strip -o mybin -f mybin.debug mybin.tmp
rm mybin.tmp

will yield a stripped mybin with embedded external debuginfo link and a detached debuginfo mybin.debug. If mybin.debug is placed in the appropriate location in /usr/lib/debug, /usr/lib/debug/.build_id, and/or on a debuginfod server, tools like dlv and gdb will automatically find its symbols when attaching to it.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 16, 2022

CC @thanm

@rsc
Copy link
Contributor

@rsc rsc commented Mar 16, 2022

Would it be sufficient to require a host linker when using this flag? Then the host linker could be in charge of writing out the archive in the standard format, and Go tools wouldn't have to know those details.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 16, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals Mar 16, 2022
@thanm
Copy link
Contributor

@thanm thanm commented Mar 17, 2022

Most applications large enough to be running into these issues (e.g. binary with DWARF is "too big") are already using external linking due to the use of CGO in a dependency somewhere. When I do a build of k8s with "ldflags=-v", I can see that the external linker is already being invoked by default for a regular build. Given that k8s is already carefully curating linker flags, it seems that the easiest path forward would be to just pass the proper options to the host linker, as Russ suggests.

@rsc
Copy link
Contributor

@rsc rsc commented Mar 23, 2022

Great. Doing it via the host linker sounds like a small amount of work.
If we did need to do the bigger amount of work for internal linking we might want to reconsider though.

@rsc rsc moved this from Active to Likely Accept in Proposals Mar 23, 2022
@rsc
Copy link
Contributor

@rsc rsc commented Mar 23, 2022

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@aarzilli
Copy link
Contributor

@aarzilli aarzilli commented Mar 23, 2022

Separate .debug files are just elf (or mach-o) files so, in theory, the linker already knows how to write them. Letting the host linker do it is of course less work. Doing .pdb is much more complicated since the format is different and (afaik) only partially documented.

@rsc rsc moved this from Likely Accept to Accepted in Proposals Mar 30, 2022
@rsc
Copy link
Contributor

@rsc rsc commented Mar 30, 2022

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: cmd/link: generate external DWARF debuginfo archives directly cmd/link: generate external DWARF debuginfo archives directly Mar 30, 2022
@rsc rsc removed this from the Proposal milestone Mar 30, 2022
@rsc rsc added this to the Backlog milestone Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Development

No branches or pull requests

6 participants