Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: generate external DWARF debuginfo archives directly #51692

Open
ringerc opened this issue Mar 16, 2022 · 12 comments
Open

cmd/link: generate external DWARF debuginfo archives directly #51692

ringerc opened this issue Mar 16, 2022 · 12 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Proposal Proposal-Accepted
Milestone

Comments

@ringerc
Copy link

ringerc commented Mar 16, 2022

TL;DR: Add an -external-debuginfo command line argument to go tool link that causes the link command to write a .debug archive containing symbol tables and DWARF debuginfo for the executable being linked.

-w and/or -s will continue to have their current effects on the output executable. These flags will have no effect on the archive produced by -external-debuginfo, which will always include the full symbol table and DWARF debuginfo.

This could be implemented by using the functionality of elfutils eu-strip in cmd/link / pkg/tool/link via libelf.

For Windows targets a .pdb file could be produced, or external debuginfo support for Windows targets could be initially omitted if it's too hard to do at the same time.

Rationale

Kubernetes project binaries and container images are currently built and distributed with the -w -s linker flags, so DWARF debuginfo is omitted then the executable is stripped of all symbols.

This saves disk space by producing smaller binaries and container images, though it has no memory consumption or performance benefit at executable runtime.

The cost is that debugging such executables is extremely difficult - only numeric addresses are available, with no symbols for functions, variables, etc. If a problem only occurs in a production or near-production system and you want to inspect the running component, you will see backtraces like:

(gdb) bt
#0  0x000000000046fc63 in ?? ()
#1  0x0000000000432fc6 in ?? ()
#2  0x00000000048e2110 in ?? ()
#3  0x0000000000000080 in ?? ()
#4  0x0000000000000000 in ?? ()

In traditional gcc or llvm based C/C++ build tool chains a balance between executable size and debuggability is maintained using external debuginfo archives. These are supported by gcc, lldb and anything using elfutils or libgdb for symbol loading. Delve supports loading of external symbols from /usr/lib/debug/.build_id and /usr/lib/debug/{path} etc, like gdb and other tools.

Neither GNU ld and LLVM ld.lld provide a command line argument for writing external debuginfo archives. Instead the release process for most binaries includes a separate pass to split the original executable with debuginfo into separate stripped executable and debuginfo archive files using objcopy --only-keep-debug, eu-strip and objcopy --add-gnu-debuglink.

The golang toolchain tries to be an integrated single-stop shop without external linker dependencies etc, so golang projects including the official k8s distribution don't tend to use this workflow. They're release-size conscious so they just don't provide any debuginfo. They just build stripped binaries for release. This is convenient but makes it much more difficult to diagnose problems.

go/link should instead support directly generating an external debuginfo archive ready for upload to a debuginfod symbol server, inclusion in a container image's /usr/lib/debug and /usr/lib/debug/.build_id trees, and/or distribution in release tarballs.

Note that delve already supports debuginfod. If go/link provided an easy way to generate external symbol archives, the k8s project or an interested 3rd party could host a public debuginfod with debuginfo for all future release binaries for use in gdb, delve, lldb, etc.

For example in gdb, one might add https://debuginfod.k8s.io/ to the DEBUGINFOD_URLS env-var, add set debuginfod enabled to .gdbinit, and have symbols automatically downloaded on demand when debugging k8s release binaries. Even over a remote gdbserver. A similar approach is possible with Delve.

Symbols can also be downloaded on-demand from a debuginfod with eu-unstrip then used as normal local detached symbol archives, and eu-make-debug-archive can be used to make a full archive of a system or container's libraries and debuginfo for remote debugging use. But the symbols must be available for that to be possible.

Interim workaround

Golang binaries can be built with debuginfo and symbols (neither -s nor -w LDFLAGS should be used) then split into separate debuginfo with eu-strip for linux targets.

This requires each build script to implement features that require dependencies that aren't part of the main go toolchain. So it's unlikely to see wide adoption until and unless the golang toolchain itself provides a standard way to do it like it has with -s for stripped executables.

But it's not overly complicated:

mv mybin mybin.tmp
eu-strip -o mybin -f mybin.debug mybin.tmp
rm mybin.tmp

will yield a stripped mybin with embedded external debuginfo link and a detached debuginfo mybin.debug. If mybin.debug is placed in the appropriate location in /usr/lib/debug, /usr/lib/debug/.build_id, and/or on a debuginfod server, tools like dlv and gdb will automatically find its symbols when attaching to it.

@ianlancetaylor
Copy link
Contributor

CC @thanm

@rsc
Copy link
Contributor

rsc commented Mar 16, 2022

Would it be sufficient to require a host linker when using this flag? Then the host linker could be in charge of writing out the archive in the standard format, and Go tools wouldn't have to know those details.

@rsc
Copy link
Contributor

rsc commented Mar 16, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals (old) Mar 16, 2022
@thanm
Copy link
Contributor

thanm commented Mar 17, 2022

Most applications large enough to be running into these issues (e.g. binary with DWARF is "too big") are already using external linking due to the use of CGO in a dependency somewhere. When I do a build of k8s with "ldflags=-v", I can see that the external linker is already being invoked by default for a regular build. Given that k8s is already carefully curating linker flags, it seems that the easiest path forward would be to just pass the proper options to the host linker, as Russ suggests.

@rsc
Copy link
Contributor

rsc commented Mar 23, 2022

Great. Doing it via the host linker sounds like a small amount of work.
If we did need to do the bigger amount of work for internal linking we might want to reconsider though.

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Mar 23, 2022
@rsc
Copy link
Contributor

rsc commented Mar 23, 2022

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@aarzilli
Copy link
Contributor

Separate .debug files are just elf (or mach-o) files so, in theory, the linker already knows how to write them. Letting the host linker do it is of course less work. Doing .pdb is much more complicated since the format is different and (afaik) only partially documented.

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Mar 30, 2022
@rsc
Copy link
Contributor

rsc commented Mar 30, 2022

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: cmd/link: generate external DWARF debuginfo archives directly cmd/link: generate external DWARF debuginfo archives directly Mar 30, 2022
@rsc rsc modified the milestones: Proposal, Backlog Mar 30, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@ringerc
Copy link
Author

ringerc commented May 16, 2023

@rsc Doing this via the host linker would be great, but as far as I can tell neither GNU ld nor llvm's lld know how to generate external debuginfo files in a single pass, so you can't just use go tool link's -extldflags to pass a flag to delegate the work to the host linker transparently.

C and C++ builds will usually compile and link an integrated binary with full debuginfo and symbols, then post-process it to produce a stripped release binary and an external debuginfo archive.

gcc and clang also have the -gsplit-dwarf flag which causes the compilation phase to emit .dwo files alongside each .o file. Then the binutils dwp command or llvm-dwp can create easily distributed DWARF-v5 debuginfo archives from the .dwo files. See https://gcc.gnu.org/wiki/DebugFission . This article has a handy demo https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/ . The result is different to a regular external debuginfo archive and requires support in the tools loading symbols, but is widely supported by ELF symbol loaders already. See also https://dwarfstd.org/issues/140421.1.html

In any case I don't see how this can be done with external linker flags alone.

I suspect that to implement this, either:

  • cmd/link would need to process the temp result binary to generate a stripped binary and optional external debuginfo in a similar manner to how binaries are split into stripped binary and external debuginfo using llvm's dsymutil or using llvm-objcopy, eu-strip or objcopy; or
  • cmd/compile would need to emit .dwo files that could be aggregated into a DWARFv5 debuginfo archive for distribution, initially using dwp or llvm-dwp, and later by a golang native tool.

It'd be good to make this easy for the golang ecosystem to adopt, so it becomes normal to publish debug symbols alongside release builds on github artifacts etc, and ideally even publish them on an elfutils debuginfod or llvm-debuginfod server for on-demand fetching. Depending on CGO makes that harder.

A lot of useful info on this topic can be found at https://maskray.me/blog/2022-10-30-distribution-of-debug-information

Sorry for the delayed response by the way. I tend to be flooded with noisy github notifications and it's hard to find the relevant ones.

@ringerc
Copy link
Author

ringerc commented May 16, 2023

@thanm Are you aware of a set of linker flags that'll produce external debuginfo in a single pass? I'd love to do it that way, at least for now, but am not aware of a way to do so.

@thanm
Copy link
Contributor

thanm commented May 16, 2023

cmd/link would need to process the temp result binary to generate a stripped binary and optional external debuginfo in >a similar manner to how binaries are split into stripped binary and external debuginfo using
llvm's dsymutil or using llvm-objcopy, eu-strip or objcopy

Yes, this is what I had in mind. In external linking the Go linker prepares an intermediate object file (e.g. "go.o") that it then hands off to the external linker as a final step. Before doing that it seems reasonable to run llvm-objcopy (or equivalent) to split off the debug info to a separate file and then pass the striped go.o to the external linker.

Implementing full split DWARF would obviously be more work. I don't think we would need to touch the compiler to do that, but it would require some changes in the linker.

@qmuntal
Copy link
Contributor

qmuntal commented May 16, 2023

I'm a little late here, just discovered this proposal.

I would prefer to keep PDB generation out of this proposal. If someday we support generating PDB files, then passing -external-debuginfo will be redundant, as PDB files can't be stored in the binary, and the user intent won't be clear: should we generate a PDB file or a split DWARF (Windows would supports both).

Having said this, would it make sense to change the flag name to -external-dwarf, or something else more DWARF-centric?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Proposal Proposal-Accepted
Projects
Status: Triage Backlog
Status: Accepted
Development

No branches or pull requests

7 participants