cmd/link: add support for automatically generating .note.gnu.build-id

[segevfiner]

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build730694799=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I want the ability to automatically generate a .note.gnu.build-id section in Go binaries. The linker currently supports a flag to get one from the outside: -B, but this is supposed to be a CRC that is calculated by the linker rather than something that can reliably be set from the outside as a linker flag.

This is used by delve and other debuggers to find split debug symbols.

What did you expect to see?

I want a flag like the binutils linker has -Wl,-build-id that adds an automatically generated .note.gnu.build-id section.

What did you see instead?

There is only a flag to set it from the outside, but it is meant to be a CRC calculated by the linker to be used reliably to differentiate different debug symbols.

[Foxboron]

I hacked a little bit on this by just running sha1 on the output buffer we have in the linker. It gives me a stable sha1 sum and I think(?) it should be good enough? Should we stamp the same build id if we use the -s -w flags maybe?

diff --git a/src/cmd/link/internal/ld/elf.go b/src/cmd/link/internal/ld/elf.go
index a1ae7eab57..3f9f115282 100644
--- a/src/cmd/link/internal/ld/elf.go
+++ b/src/cmd/link/internal/ld/elf.go
@@ -10,6 +10,7 @@ import (
        "cmd/internal/sys"
        "cmd/link/internal/loader"
        "cmd/link/internal/sym"
+       "crypto/sha1"
        "debug/elf"
        "encoding/binary"
        "encoding/hex"
@@ -825,6 +826,11 @@ func elfgobuildid(sh *ElfShdr, startva uint64, resoff uint64) int {
        return elfnote(sh, startva, resoff, n)
 }

+func elfproducebuildinfo(out *OutBuf) {
+       id := sha1.Sum(out.Data())
+       buildinfo = id[:]
+}
+
 func elfwritebuildinfo(out *OutBuf) int {
        sh := elfwritenotehdr(out, ".note.gnu.build-id", ELF_NOTE_BUILDINFO_NAMESZ, uint32(len(buildinfo)), ELF_NOTE_BUILDINFO_TAG)
        if sh == nil {
@@ -1396,9 +1402,7 @@ func (ctxt *Link) doelf() {
        if ctxt.IsFreebsd() {
                shstrtab.Addstring(".note.tag")
        }
-       if len(buildinfo) > 0 {
-               shstrtab.Addstring(".note.gnu.build-id")
-       }
+       shstrtab.Addstring(".note.gnu.build-id")
        if *flagBuildid != "" {
                shstrtab.Addstring(".note.go.buildid")
        }
@@ -1911,6 +1915,9 @@ func asmbElf(ctxt *Link) {
                phsh(pnotei, sh)
        }

+       if len(buildinfo) == 0 {
+               elfproducebuildinfo(ctxt.Out)
+       }
        if len(buildinfo) > 0 {
                sh := elfshname(".note.gnu.build-id")
                resoff -= int64(elfbuildinfo(sh, uint64(startva), uint64(resoff)))
@@ -2285,9 +2292,7 @@ elfobj:
                if ctxt.HeadType == objabi.Hfreebsd {
                        a += int64(elfwritefreebsdsig(ctxt.Out))
                }
-               if len(buildinfo) > 0 {
-                       a += int64(elfwritebuildinfo(ctxt.Out))
-               }
+               a += int64(elfwritebuildinfo(ctxt.Out))
                if *flagBuildid != "" {
                        a += int64(elfwritegobuildid(ctxt.Out))
                }

[Foxboron]

@bcmills sorry for the ping here. But has there been any discussions on how to solve this?

Would a sha256 hash over the actionID and/or the .note.go.buildid string be a good enough solution?

Generally this would help debug packages on Linux distros and debuggers like delve do the right thing instead of having to pass the -buildid linker flag explicitly.

[ianlancetaylor]

We already have a Go build ID. We can just write that out as the GNU build ID as well. I think the only question is whether we should always do it. The GNU linker only does it upon request, but on many Linux systems the compiler is configured to request it by default (by passing a --build-id option when invoking the linker).

[Foxboron]

Most (all?) debuggers expect the value for basic functionality and I struggle to see why we would stamp go build id and not the GNU build id along with it?

The only issue is that the GNU build ID needs to either be a md5 or sha1, so I don't think we can pass the Go build id as-is?

Having this as a build option for distributions is non-trivial as it's still hard to override the command line arguments passed to the compiler.

[ianlancetaylor]

To the best of my knowledge the GNU build ID does not have to be an MD5 or SHA1. Those are just common implementations of it. It can be any random relatively unique string.

[ianlancetaylor]

For example, for the gold linker the default value (if you just specify --build-id with no specified style) is neither MD5 nor SHA1, it's a tree hash that can be computed in parallel.

[prattmic]

Linux's perf tool is one such tool that automatically records the GNU buildid (directly in the kernel during mmap event generation), which can then be used by downstream tooling to find the correct binary for symbolization. For pure Go binaries, no build ID is recorded.

[prattmic]

We already have a Go build ID. We can just write that out as the GNU build ID as well. I think the only question is whether we should always do it.

FWIW, Linux perf expects the GNU build ID to be 20 bytes or less (https://elixir.bootlin.com/linux/v6.2/source/lib/buildid.c#L30). I'm not sure if this is specified or a convention, but the Go build ID is significantly longer (83 bytes).

[znkr]

Truncating larger hashes seems to be okay: https://reviews.llvm.org/D121531