Go and llvm > 14.0.0

[rymurr]

Hi all,

I was curious about the comment here: https://github.com/grailbio/bazel-toolchain/blob/master/tests/WORKSPACE#L60

I've been hitting this w/ 15.0.6 and getting hard crashes in simple-ish go programs. Do you know what the issue is and why >14.0.0 doesn't work?

[siddharthab]

I have not had time to look into this. If you can reproduce this outside of bazel with a simple example, maybe file a bug with the Go team. I can look into this more if it is only happening when using this toolchain.

[garymm]

To add context, the errors I'm seeing are only on Linux and happen at runtime when running go_tests that have cgo dependencies. The error messages are like:

runtime: pcHeader: magic= 0xfffffff0 pad1= 0 pad2= 0 minLC= 1 ptrSize= 8 pcHeader.textStart= 0x2196a40 text= 0x55b57c8d5a40 pluginpath= 
fatal error: invalid function symbol table
runtime: panic before malloc heap initialized

And I'm seeing this with LLVM 15.0.6.

[garymm]

Looks like maybe the problem is -s being set when linking?
golang/go#6245

Is there any code in this repo that controls that flag?
Or maybe that flag behaves differently for newer LLVM?

[siddharthab]

I think I would need an example cgo program compiled with LLVM 15.0.6 that runs to completion when not using this bazel toolchain configuration, but crashes when we use it. I can backtrack from there to see what we are doing different in this toolchain.

[siddharthab]

Looks like with Go 1.20, even LLVM 14 is not working. I was also able to reproduce the original issue with LLVM 15. I will try to look into this at some point.

[siddharthab]

Was able to give a little bit of time today to this. I suspect this is because we use LLD and not gold. When I switched to gold in this repo, the Go targets worked fine. I will work on this a little more to ensure we make it easy for people to opt out of LLD.

[siddharthab]

A few more hints. LLD in general is fine, but not with -pie. And somehow with this project's toolchain, rules_go is setting forcing PIC, but the standard toolchain does not.

Was able to reproduce the crashes on a pristine rules_go repo with:

sudo apt-get install clang-15 lld-15
git clone git@github.com:bazelbuild/rules_go.git && cd rules_go
CC=clang-15 bazel run --force_pic //tests/core/cgo:c_srcs

[siddharthab]

c217c03 shows that --linkopt=-nopie is a workaround.

[rymurr]

thanks @siddharthab ! I tried real quick and thinks run a lot better. I am happy to use the workaround.

[garymm]

If someone can repro this without bazel (just with the go test command) then I guess that there's just some incompatibility between Go + LLD + PIE.

[siddharthab]

Standard Go toolchain works fine with -fPIC and -pie and LLD. I am going to open an issue in rules_go because I think it has to do with how they make stdlib.

[siddharthab]

OK so if we are forcing PIC, then linkmode needs to be set as "pie" in the go_binary target to set the other flags correctly. I think it makes sense. I think it may be a coincidence that this problem does not happen with gold.

I submitted bazel-contrib/rules_go#3691 with more details, but I don't think there is much to do here.

I will now check why rules_go is picking up -pie from this toolchain to add to -extldflags.

[siddharthab]

Alright, I have the full story now.

LLVM switched to always building with PIE, starting with v15; gcc had been doing it for some time.
https://discourse.llvm.org/t/llvm-15-default-pie-issue/67125/3

Go toolchain, knowing this, actively provides the --no-pie flag to the linker.
https://github.com/golang/go/blob/495830acd6976c8a2b39dd4aa4fdc105ad72de52/src/cmd/link/internal/ld/lib.go#L1815

But to check if the flag is accepted by the linker, it first runs a dummy linker command in a temporary directory. Which means relative paths to the linker will not work. This will be a problem for any hermetic toolchain like ours.
Good news: this has already been fixed upstream in golang/go#59952.

Sent bazel-contrib/rules_go#3692 for supporting Go versions older than 1.21.