cmd/link: missing CGO symbols in `.dynsym` on Go >= 1.21 on arm64

[mtardy]

What version of Go are you using (go version)?

$ go version
go version go1.21.1 linux/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/mahe.linux/.cache/go-build'
GOENV='/home/mahe.linux/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/mahe.linux/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/mahe.linux/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1359563043=/tmp/go-build -gno-record-gcc-switches'

What did you do?

So the issue is a difference of behavior from Go 1.20 to Go 1.21 on arm64 (and not on amd64).

Let's make a minimal reproducing setup:

1. Put that in a file, for example main.go and go build main.

   //go:build linux
   // +build linux
   
   package main
   
   /*
   int symbol_test(void)
   {
           return 0;
   }
   */
   import "C"
   
   func main() {
           C.symbol_test()
   }

2. Use a binary to read the symbols, for example readelf -s main | grep symbol_test, you can use less as well to see:

   • With Go 1.20.8 (and before), there is a symbol_test in .dynsym and symbol_test in .symtab.
   • With Go 1.21.0 and 1.21.1, there is only a symbol_test in .symtab, it was removed from .dynsym.

So I cloned Go and did a git bissect to see which commit was the culprit and found that it was 1f29f39 -> https://go-review.googlesource.com/c/go/+/414654. Unfortunately, I lack context and understanding of the Go codebase to understand why this changed the behavior like that on arm64 specifically and if this is unintended.

For context, I was using uprobes on a Go binary, and for a reason that is not entirely clear for now, the lib I was using to load the probe is only reading .dynsym.

What did you expect to see?

I expected this behavior not to change, and especially not to change only on arm64 while amd64 is still behaving the same as in the past.

What did you see instead?

I saw that the symbol was missing in .dynsym using Go >= 1.21.

[seankhliao]

cc @ianlancetaylor

[cherrymui]

Thanks for the report. Previously we dynamically export ~all symbols, which is too many and unnecessary for many of them. With that CL, we only export Go symbols that are dynamically exported to C. I think we probably should also export C symbols (that are not marked as static or hidden visibility) per C semantics. @ianlancetaylor what do you think?

[cherrymui]

on arm64 (and not on amd64)

This is interesting. I don't think that behavior depends on the architecture. Maybe this could be due to different C toolchain versions? What C compiler and linker do you use for both architectures?

[mtardy]

on arm64 (and not on amd64)

This is interesting. I don't think that behavior depends on the architecture. Maybe this could be due to different C toolchain versions? What C compiler and linker do you use for both architectures?

Yes that may be the reason. Initially, I uncovered that new thing because in the PR to update to Go 1.21 for the project I'm working on cilium/tetragon#1344, we had only the Go test failing on arm64. See this run for example https://github.com/cilium/tetragon/actions/runs/6119884977/job/16610682208?pr=1344. The reason why it failed is because we load a uprobe at some point and a symbol that we previously used was missing only on the arm64 runners binaries. But those are GitHub-hosted runners for x86_64 and BuildJet self-hosted runners for arm64, so they might have different toolchains installed indeed (and I can't tell you exactly what's running on those without running workflows for that). I could however reproduce easily the issue to isolate the commit on my arm64 machine.

[ianlancetaylor]

The C linker doesn't export all symbols to the dynamic symbol table by default. I think that people who rely on this should be passing -extldflags=-Wl,--export-all-symbols to go build. It should also work to add this to a #cgo LDFLAGS line in a cgo comment. Instead of --export-all-symbols you could of course use --export-dynamic-symbol for particular symbols.

In general CL 414654 made the Go linker behave more like a C linker with regard to when dynamic symbols are exported. I think that is a good thing and I don't yet see a reason to change that. I agree that it is a change in behavior.

[mtardy]

So I just understood something new, my issue originally came from the fact that when compiling and running with go test (without the -c flag), copying the binary and readelf(ing) will end up with a binary missing the .symtab (thus being the only one containing the symbol in Go 1.21, which means the symbol cannot be found), only .dynsym is there. When I do go test -c, the .symtab is there, and the test passes because I can find the symbol (in .symtab).

Basically, when I do go test, I end up with a stripped binary, and with go test -c the binary is not stripped. I thought those were equivalent and go test was just a shortcut to compiling and running.

I'm not very familiar with how behaves go test but why do I have this difference?

[cherrymui]

go test without -c generates a one-off binary that is supposed to immediately run and get deleted, so it by default suppresses symbol table, debug info, etc. to make build faster. It is not a binary that you want to copy or hold for it. If you need a binary to investigate, use go test -c, which generates a binary with symbol table and debug info.

[mtardy]

go test without -c generates a one-off binary that is supposed to immediately run and get deleted, so it by default suppresses symbol table, debug info, etc. to make build faster. It is not a binary that you want to copy or hold for it. If you need a binary to investigate, use go test -c, which generates a binary with symbol table and debug info.

ah yeah ok I see. I end up with a different behavior since I'm using the binary itself to hook a uprobe and then it can now fail since the symbol is now absent from .dynsym and the whole .symtab can be removed as well. But I should probably not do that and rely on a separate program, thanks for all your answers :)

[cherrymui]

Thanks. Is there anything still needed for this issue? Or we can close it? Thanks.

[mtardy]

Thanks. Is there anything still needed for this issue? Or we can close it? Thanks.

Cool! I guess no regarding this response:

In general CL 414654 made the Go linker behave more like a C linker with regard to when dynamic symbols are exported. I think that is a good thing and I don't yet see a reason to change that. I agree that it is a change in behavior.

If the change of behavior is okay/better on your side, I just need to adapt and it won't be hard. I was just thinking that it might have been unexpected for the Go project and was interested in it.

I was going to close it but I'll let you close it as not planned or completed, whatever you prefer.