Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: -buildmode=pie does not build correct PIE binaries for ppc64le #28531

Open
laboger opened this issue Nov 1, 2018 · 31 comments
Milestone

Comments

@laboger
Copy link
Contributor

@laboger laboger commented Nov 1, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

latest

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

ppc64le

What did you do?

Tried to run binaries built as PIE.

What did you expect to see?

Correct execution

What did you see instead?

Error messages like this:
./main.exe: error while loading shared libraries: R_PPC64_ADDR16_HA re10f2f8fa4 for symbol `' out of range
2018/09/05 09:53:14 Failed: exit status 127
on newer distros.

The problem is related to issues #27510 and #21954 and can still be reproduced on many distros by passing -pie as a linker option.

The solution to the above issues was to use -buildmode=pie but that doesn't correct all problems. When linking a program with -pie on ppc64le then all code should be built as PIC. When some code is not PIC there could be relocations created like R_PPC64_ADDR16_HA which will fail with the above error if the relocated addresses are too big. On some distros/kernels we get lucky and no failures occur if the addresses stay small enough but on newer distros the runtime relocation errors can occur.

When using -buildmode=pie the -shared option is set to force the compiles to be built as PIC but it still links in stdlib which was not built as PIC. Likewise any library linked in should be PIC. When all the code is has been built as PIC then this relocation would not be generated and the PIE should work.

One further question is whether all code should be PIC if the gcc being used for the link passes -pie by default and stop setting -no-pie.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 1, 2018

When you use -buildmode=pie the go tool is supposed to rebuild all dependencies using -shared, including the standard library. If that is not happening, something is wrong.

As far as a gcc defaulting to PIE, we already have support when using -compiler=gccgo for passing -fno-pie; perhaps we need to extend that support to the gc compiler side. See gccNoPie in cmd/go/internal/work/.

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Nov 1, 2018
@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Nov 1, 2018

I'm sorry there's some confusion here. This was reported to me through a bugzilla that even with -buildmode=pie a bad relocation is generated but I was mistaken above about whether stdlib is built with PIC in this case. In my experiments it seems to be. The reported failure is on RHEL8 which I don't have access to so I'm not sure yet where it is coming from.

One related question is whether golang should recognize whether the gcc on the system sets -pie by default when invoking the linker and if so just build it as -buildmode=pie. I'm not sure if that's the right answer but just what was suggested.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 1, 2018

I don't think we should change the default of go build to the default of GCC. That seems like an unnecessary tying that will tend to increase confusion.

Perhaps we should have an easy way for a distro to change the default of go build, but that seems like a different issue. (For example, we could consider adding another environment variable recognized by src/make.bash.)

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Nov 5, 2018

@ianlancetaylor Is there a way to build a toolchain for ppc64le so that the default type of executable is always PIE through the use of existing environment variables or other options? Or is it more complicated than that?

I tried to just change the default in cmd/go/internal/work/init.go as a test to see what would happen and got this:

Building Go toolchain2 using go_bootstrap and Go toolchain1.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
Building Go toolchain3 using go_bootstrap and Go toolchain2.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel +745ec8b Fri Nov 2 17:56:07 2018 +0000"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch ppc64le\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "compile ymJr-uldWHMJ1NKhKQaW X:framepointer [\"-shared\"] []\n"
HASH[build runtime/internal/sys]: "GO$GOARCH=\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch_ppc64le.go: d43466ea44c56eba4514ad41e6652168a0ce0dcb720578ea6dadea343266a24a
HASH[build runtime/internal/sys]: "file arch_ppc64le.go 1DRm6kTFbrpFFK1B5mUh\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/stubs.go: 376cb070d0d0d7094a8a5ed372853e00c377c2aba3d8c118dee821dd526a6022
HASH[build runtime/internal/sys]: "file stubs.go N2ywcNDQ1wlKil7TcoU-\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoarch_ppc64le.go: 157322cbcc4e3c5d80effef8a81d3d9a480cc2558550fad0df46a10b48af334c
HASH[build runtime/internal/sys]: "file zgoarch_ppc64le.go FXMiy8xOPF2A7_74qB09\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoos_linux.go: 32284a5a70366b56fe207ea3a16ad5d61dd208d425ad28b81cc2fe59db9180e8
HASH[build runtime/internal/sys]: "file zgoos_linux.go MihKWnA2a1b-IH6joWrV\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zversion.go: a0901e5adb74da30a6a6643ab36cfaa28c4c5136142e9a1fdeec8ca6c30f000a
HASH[build runtime/internal/sys]: "file zversion.go oJAeWtt02jCmpmQ6s2z6\n"
HASH[build runtime/internal/sys]: 664409261b5745f2e0926a0ef91599e67e2adb4a49da5a0b03eef7256f034c44
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /home/boger/golang/clean/go/pkg/tool/linux_ppc64le/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 5, 2018

There is no simple way today to force the gc toolchain to default to PIE.

I'm not sure why changing buildModeInit didn't work. It looks like it got confused with the cache. Maybe some extra step is required to add the different build mode to the cache key somehow.

@jfkw

This comment has been minimized.

Copy link

@jfkw jfkw commented Dec 13, 2018

This issue is also present on SUSE and openSUSE for ppc64le.

With go1.10.5, the build stops at the following error:

cmd/vendor/github.com/google/pprof/internal/symbolizer
cmd/vendor/github.com/google/pprof/internal/driver
cmd/vendor/github.com/google/pprof/driver
cmd/pprof
/home/abuild/rpmbuild/BUILD/go/bin/go: error while loading shared libraries: R_PPC64_ADDR16_HA re10143fb0c for symbol `' out of range

By ensuring gcc does not use PIE by default, the build can complete, although this is not ideal.

It would be a great improvement to have an environment variable and option to force the toolchain to default to PIE.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Dec 13, 2018

@jfkw Now in 1.12 you should be able to use GOFLAGS=-buildmode=pie.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Jan 17, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@daixiang0

This comment has been minimized.

Copy link

@daixiang0 daixiang0 commented Oct 16, 2019

@ianlancetaylor seems not work for 1.12:

# go version
go version go1.12.10 linux/mips64le
# GOFLAGS=-buildmode=pie go build components/cli
-buildmode=pie not supported on linux/mips64le
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 16, 2019

@daixiang0 This bug report is about a problem with ppc64le. You are using mips64le.

@daixiang0

This comment has been minimized.

Copy link

@daixiang0 daixiang0 commented Oct 17, 2019

@ianlancetaylor sorry for mistake, BTW any plan for mips to support pie build mode?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 17, 2019

@daixiang0 Sorry, I do not know of anyone working on adding support for PIE for MIPS.

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

I believe the title is is misleading. -buildmode=pie works fine for most things on ppc64le. The problem happens when trying to build a toolchain that will generate all binaries as pie by default, including those that are part of the toolchain. This came about because of a request from a single user.

No one has asked about this again, so if we want to close it that is fine with me.

I retried the same experiment as I did back in November and got the same result.

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 17, 2019

So now you have to add 'me too' to bugs? It his some web 2.0 thing?

Anyway, the code is just broken.

How do you suppose the code is position independent if the libraries included in the interpreter are not?

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 17, 2019

C or C++ code built for ppc64le should be position independent by default.

Apparently not. Some distributions choose to set this default which breaks go completely.

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

I'm very sorry. I didn't notice that an earlier comment was made regarding this topic related to SUSE.
@ianlancetaylor I tried a build setting GOFLAGS=-buildmode=pie, and the build finished but that didn't seem to build all the toolchain binaries as PIE.

If I change the default in init.go, I still get the errors that some files are stale.

I cannot reproduce the problem identified earlier on SUSE using go 1.10.5. Were there any env variables used during the build? What distro was it.

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 17, 2019

In openSUSE there is a package gcc-PIE which sets the gcc defaults options to PIC and PIE (since Leap 15.0 which is the oldest version still developed). They can still be overridden on the commandline or the package may opt to not install these defaults.

Not using the PIE options was what was done in openSUSE to build go and go packages on ppc64le.
I can look up the current state if there is some interest in debugging this further.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 20, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 20, 2019

@hramrach Please be polite when making comments, and bear in mind the Gopher Code of Conduct (https://golang.org/conduct). In particular, be patient and be charitable. Thanks.

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 20, 2019

What the R_PPC64_ADDR16_HA suggests is you generate direct references which are filled in as immediates in the code which are only 16bit and you need to generate indirect jumps or loads to be able to link non-trivial ppc binaries referencing code or data more than +-32kb away.

@jfkw

This comment has been minimized.

Copy link

@jfkw jfkw commented Oct 20, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

@ianlancetaylor Currently, the openSUSE ppc64le go package builds by not installing the gcc-PIE package. I will do some testing in openSUSE ppc64le using the GOFLAGS=-buildmode=pie used uniformly for building go, go tools and small programs written in go, and report back here in the coming weeks to correlate results with the observations from @laboger.

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 21, 2019

I suspect this is a bootstrap issue to some extent. You cannot build pie binaries if your stdlib is not PIC. You probably need to rebuild twice, once to build PIC stdlib injecting the PIC flag somewhere, and second time to build the PIE binaries.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach If I understand you correctly, then the go tool should consistently do the right thing. Using go build -buildmode=pie will build all Go code, including the standard library, as appropriate for PIE. Using go build without -buildmode=pie will build all Go code, including the standard library, as appropriate for ordinary executables.

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 21, 2019

that's not possible in the case when you are trying to build a pie compiler and all you have is non-PIE compiler with non-PIC stdlib. The new compiler will use the old stdlib which is not suitable. So you first need to build a compiler with PIC stdlib which does not default to PIE and then fully PIE compiler. Assuming the code generation works and correct relocations are generated.

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Oct 21, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

Yes, if I use -x I can see it uses -buildmode=pie on the final link of the executable for hello.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach I'm sorry, I'm not sure what you mean. When you say "stdlib", what are you referring to? When you say "interpreter", do you mean the ELF interpreter, also known as the dynamic linker?

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 21, 2019

sorry, I meant compiler. Looks like interpreter is not standard part of go.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach When you say "stdlib", what are you referring to?

@hramrach

This comment has been minimized.

Copy link

@hramrach hramrach commented Oct 21, 2019

whatever comes with the compiler that might be linked into binaries.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

Assuming you mean the Go compiler, then, as I explained at #28531 (comment), the Go standard library will be rebuilt, if necessary, with the options required to support the requested build mode. This happens automatically and does not require any user intervention. The results are cached to avoid excess rebuilding.

@laboger

This comment has been minimized.

Copy link
Contributor Author

@laboger laboger commented Oct 21, 2019

After I do a normal build, I can do this to make these binaries built as pie:

go install -buildmode=pie cmd/go
go install -buildmode=pie cmd/trace
go install -buildmode=pie cmd/pprof

Does this help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.