Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: invalid TLS access model used with -buildmode c-shared #53214

Open
aadamowski opened this issue Jun 2, 2022 · 7 comments
Open

cmd/link: invalid TLS access model used with -buildmode c-shared #53214

aadamowski opened this issue Jun 2, 2022 · 7 comments
Labels
NeedsInvestigation WaitingForInfo

Comments

@aadamowski
Copy link

@aadamowski aadamowski commented Jun 2, 2022

What version of Go are you using (go version)?

$ link -V
link version go1.18

Does this issue reproduce with the latest release?

Yes, 1.18 is the latest release.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/olo/.cache/go-build"
GOENV="/home/olo/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/olo/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/olo/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3022165828=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We have a CGo-based library that we build in c-shared mode to use as a dynamic shared object on Linux (x86_64).

The library is, unfortunately, an internal project and I cannot disclose the source code.

For the C compiler, we are using Clang.

The final linking of the shared object file is performed with the following invocation of the go link tool (some irrelevant flags removed):

go link -s -X internalmodule/go/buildinfo.Compiler=clang -o shared_library.so -buildmode c-shared -buildid= -linkmode external -L importdir -extld /path/to/external_clang++_ld  '-extldflags=A BUNCH OF EXT LD FLAGS' package.a

What did you expect to see?

All this setup has been working fine with Clang's previous version, 9.0.20190721 and the exact same (binary for binary) Golang toolchain version.

We would expect it to work the same after upgrading to a newer Clang version.

What did you see instead?

With a newer Clang version 12.0.20210610, we are observing cmd/link fail with the following error:

ld.lld: error: relocation R_X86_64_TPOFF32 against runtime.tlsg cannot be used with -shared
>>> defined in /tmp/go-link-4046235013/go.o
>>> referenced by cpu_x86.go:65 (/usr/local/go/src/internal/cpu/cpu_x86.go:65)
>>>               /tmp/go-link-4046235013/go.o:(internal/cpu.doinit)

The first suspicion would obviously be on Clang. However, we tracked it down to this change: https://reviews.llvm.org/D33100

As can be seen, Clang has introduced a stricter check on where this relocation is allowed due to correctness reasons.

Further reading of https://akkadia.org/drepper/tls.pdf indicates that this relocation type is intended for the TLS Local Exec model, and exec models are not suitable for usage in dynamic shared objects (only in final built executables).

We've analyzed all the intermediate/dependency .a archive files and the .o object files they contain with readelf -r, and none of them contain the R_X86_64_TPOFF32 relocation anywhere.

The temporary go.o file generated by the go link command is the first that appears during the build process that contains it, indicating that it is this command inserting the relocation.

We've tracked down the code in cmd/link that generates this relocation to this place:
https://github.com/golang/go/blob/go1.18/src/cmd/link/internal/amd64/asm.go#L403

which further confirms that cmd/link is using an invalid TLS access model for the situation (-buildmode c-shared was used).

I do not yet have a good enough understanding of the Go linker and why is it choosing this access model, but its behavior seems to be clearly incorrect.

It's possible that, as the latest Clang releases get more prevalent, this issue will pop up in other places with other users who rely on -buildmode c-shared.

This issue might also have some relation to issue #48596 and bears some similarity to the old (and fixed) issue #9652.

@seankhliao seankhliao added the NeedsInvestigation label Jun 3, 2022
@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Jun 3, 2022

cc @cherrymui @rsc @ianlancetaylor

@cherrymui
Copy link
Member

@cherrymui cherrymui commented Jun 6, 2022

Thanks for the issue. Could you share how you invoke the build commands? Do you run go link (by which I think you mean go tool link) directly? Do you use go build -buildmode c-shared?

@cherrymui cherrymui added the WaitingForInfo label Jun 6, 2022
@aadamowski
Copy link
Author

@aadamowski aadamowski commented Jun 23, 2022

Hi! Yes to both of those things (we use the Buck build system which breaks down the process into individual invocations of Go tools like compile, link, asm or pack directly). We supply -buildmode c-shared to link.

I'm currently working on a minimalized repro case that only relies on pure Golang toolchain, but other things will keep me busy till about mid-July, so cannot provide more info at the moment.

@cherrymui
Copy link
Member

@cherrymui cherrymui commented Jun 23, 2022

What flags do you pass to compile and asm? Could you show a full list of commands you run? Thanks.

Note that not all compiler/assembler flags are compatible with the build mode.

@aadamowski
Copy link
Author

@aadamowski aadamowski commented Jun 23, 2022

The original build is quite extensive and complicated, but here's the gist of what's being run to produce some of the key .o's that feed into the problematic link:

compile -p main -pack -trimpath <ROOT_OF_SCM_REPO> -nolocalimports -dynlink '-buildid=' -o <OUTPUT_ARCHIVE>.a -I <INCLUDE_DIR> -complete <ADDITIONAL_BUILD_INFO_SYMBOLS>.go /PATH/TO/main.go

The asm invocation is only used on some of the Golang core crypto lib internals way earlier in the build dependency tree, not on any of our own sources:

asm -trimpath <ROOT_OF_SCM_REPO> -D GOOS_linux -D GOARCH_amd64 -o <OUTPUT_OBJECT>.o -I <INCLUDE_DIR> -I <golang.org/x/crypto/curve25519/internal/field/field_HEADERS_DIR> -I <golang.org/x/crypto/curve25519/internal/field/field_HEADERS_DIR> <GOLANG_SOURCE_TREE>/go/golang.org/x/crypto/curve25519/internal/field/fe_amd64.s

I believe this part is a recreation of Golang's own distribution build within Buck.

Is there perhaps some key arg to go compile that we aren't supplying that causes its produced .o to trigger go link switch to logic that chooses the LocalExec Thread Local Storage access model?

@cherrymui
Copy link
Member

@cherrymui cherrymui commented Jun 23, 2022

I think you need to pass the -shared flag to the compiler and the assembler. I don't think you want to pass the -dynlink flag.

Note that this also applies to the Go standard library packages. If you use pre-built packages for the standard library, make sure they are built with those flags as well.

@aadamowski
Copy link
Author

@aadamowski aadamowski commented Jun 23, 2022

Thanks for the suggestion! I'll try that and report back. It might need to wait till August, though, due to other things I need to work on first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation WaitingForInfo
Projects
None yet
Development

No branches or pull requests

3 participants