Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: -buildmode=pie does not build correct PIE binaries for ppc64le #28531

Open
laboger opened this issue Nov 1, 2018 · 38 comments
Open

cmd/go: -buildmode=pie does not build correct PIE binaries for ppc64le #28531

laboger opened this issue Nov 1, 2018 · 38 comments

Comments

@laboger
Copy link
Contributor

@laboger laboger commented Nov 1, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

latest

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

ppc64le

What did you do?

Tried to run binaries built as PIE.

What did you expect to see?

Correct execution

What did you see instead?

Error messages like this:
./main.exe: error while loading shared libraries: R_PPC64_ADDR16_HA re10f2f8fa4 for symbol `' out of range
2018/09/05 09:53:14 Failed: exit status 127
on newer distros.

The problem is related to issues #27510 and #21954 and can still be reproduced on many distros by passing -pie as a linker option.

The solution to the above issues was to use -buildmode=pie but that doesn't correct all problems. When linking a program with -pie on ppc64le then all code should be built as PIC. When some code is not PIC there could be relocations created like R_PPC64_ADDR16_HA which will fail with the above error if the relocated addresses are too big. On some distros/kernels we get lucky and no failures occur if the addresses stay small enough but on newer distros the runtime relocation errors can occur.

When using -buildmode=pie the -shared option is set to force the compiles to be built as PIC but it still links in stdlib which was not built as PIC. Likewise any library linked in should be PIC. When all the code is has been built as PIC then this relocation would not be generated and the PIE should work.

One further question is whether all code should be PIC if the gcc being used for the link passes -pie by default and stop setting -no-pie.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 1, 2018

When you use -buildmode=pie the go tool is supposed to rebuild all dependencies using -shared, including the standard library. If that is not happening, something is wrong.

As far as a gcc defaulting to PIE, we already have support when using -compiler=gccgo for passing -fno-pie; perhaps we need to extend that support to the gc compiler side. See gccNoPie in cmd/go/internal/work/.

@laboger
Copy link
Contributor Author

@laboger laboger commented Nov 1, 2018

I'm sorry there's some confusion here. This was reported to me through a bugzilla that even with -buildmode=pie a bad relocation is generated but I was mistaken above about whether stdlib is built with PIC in this case. In my experiments it seems to be. The reported failure is on RHEL8 which I don't have access to so I'm not sure yet where it is coming from.

One related question is whether golang should recognize whether the gcc on the system sets -pie by default when invoking the linker and if so just build it as -buildmode=pie. I'm not sure if that's the right answer but just what was suggested.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 1, 2018

I don't think we should change the default of go build to the default of GCC. That seems like an unnecessary tying that will tend to increase confusion.

Perhaps we should have an easy way for a distro to change the default of go build, but that seems like a different issue. (For example, we could consider adding another environment variable recognized by src/make.bash.)

@laboger
Copy link
Contributor Author

@laboger laboger commented Nov 5, 2018

@ianlancetaylor Is there a way to build a toolchain for ppc64le so that the default type of executable is always PIE through the use of existing environment variables or other options? Or is it more complicated than that?

I tried to just change the default in cmd/go/internal/work/init.go as a test to see what would happen and got this:

Building Go toolchain2 using go_bootstrap and Go toolchain1.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
Building Go toolchain3 using go_bootstrap and Go toolchain2.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel +745ec8b Fri Nov 2 17:56:07 2018 +0000"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch ppc64le\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "compile ymJr-uldWHMJ1NKhKQaW X:framepointer [\"-shared\"] []\n"
HASH[build runtime/internal/sys]: "GO$GOARCH=\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch_ppc64le.go: d43466ea44c56eba4514ad41e6652168a0ce0dcb720578ea6dadea343266a24a
HASH[build runtime/internal/sys]: "file arch_ppc64le.go 1DRm6kTFbrpFFK1B5mUh\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/stubs.go: 376cb070d0d0d7094a8a5ed372853e00c377c2aba3d8c118dee821dd526a6022
HASH[build runtime/internal/sys]: "file stubs.go N2ywcNDQ1wlKil7TcoU-\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoarch_ppc64le.go: 157322cbcc4e3c5d80effef8a81d3d9a480cc2558550fad0df46a10b48af334c
HASH[build runtime/internal/sys]: "file zgoarch_ppc64le.go FXMiy8xOPF2A7_74qB09\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoos_linux.go: 32284a5a70366b56fe207ea3a16ad5d61dd208d425ad28b81cc2fe59db9180e8
HASH[build runtime/internal/sys]: "file zgoos_linux.go MihKWnA2a1b-IH6joWrV\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zversion.go: a0901e5adb74da30a6a6643ab36cfaa28c4c5136142e9a1fdeec8ca6c30f000a
HASH[build runtime/internal/sys]: "file zversion.go oJAeWtt02jCmpmQ6s2z6\n"
HASH[build runtime/internal/sys]: 664409261b5745f2e0926a0ef91599e67e2adb4a49da5a0b03eef7256f034c44
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /home/boger/golang/clean/go/pkg/tool/linux_ppc64le/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 5, 2018

There is no simple way today to force the gc toolchain to default to PIE.

I'm not sure why changing buildModeInit didn't work. It looks like it got confused with the cache. Maybe some extra step is required to add the different build mode to the cache key somehow.

@jfkw
Copy link

@jfkw jfkw commented Dec 13, 2018

This issue is also present on SUSE and openSUSE for ppc64le.

With go1.10.5, the build stops at the following error:

cmd/vendor/github.com/google/pprof/internal/symbolizer
cmd/vendor/github.com/google/pprof/internal/driver
cmd/vendor/github.com/google/pprof/driver
cmd/pprof
/home/abuild/rpmbuild/BUILD/go/bin/go: error while loading shared libraries: R_PPC64_ADDR16_HA re10143fb0c for symbol `' out of range

By ensuring gcc does not use PIE by default, the build can complete, although this is not ideal.

It would be a great improvement to have an environment variable and option to force the toolchain to default to PIE.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Dec 13, 2018

@jfkw Now in 1.12 you should be able to use GOFLAGS=-buildmode=pie.

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Jan 17, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@daixiang0
Copy link

@daixiang0 daixiang0 commented Oct 16, 2019

@ianlancetaylor seems not work for 1.12:

# go version
go version go1.12.10 linux/mips64le
# GOFLAGS=-buildmode=pie go build components/cli
-buildmode=pie not supported on linux/mips64le
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 16, 2019

@daixiang0 This bug report is about a problem with ppc64le. You are using mips64le.

@daixiang0
Copy link

@daixiang0 daixiang0 commented Oct 17, 2019

@ianlancetaylor sorry for mistake, BTW any plan for mips to support pie build mode?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 17, 2019

@daixiang0 Sorry, I do not know of anyone working on adding support for PIE for MIPS.

@laboger
Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

I believe the title is is misleading. -buildmode=pie works fine for most things on ppc64le. The problem happens when trying to build a toolchain that will generate all binaries as pie by default, including those that are part of the toolchain. This came about because of a request from a single user.

No one has asked about this again, so if we want to close it that is fine with me.

I retried the same experiment as I did back in November and got the same result.

@hramrach
Copy link

@hramrach hramrach commented Oct 17, 2019

So now you have to add 'me too' to bugs? It his some web 2.0 thing?

Anyway, the code is just broken.

How do you suppose the code is position independent if the libraries included in the interpreter are not?

@laboger
Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

@hramrach
Copy link

@hramrach hramrach commented Oct 17, 2019

C or C++ code built for ppc64le should be position independent by default.

Apparently not. Some distributions choose to set this default which breaks go completely.

@laboger
Copy link
Contributor Author

@laboger laboger commented Oct 17, 2019

I'm very sorry. I didn't notice that an earlier comment was made regarding this topic related to SUSE.
@ianlancetaylor I tried a build setting GOFLAGS=-buildmode=pie, and the build finished but that didn't seem to build all the toolchain binaries as PIE.

If I change the default in init.go, I still get the errors that some files are stale.

I cannot reproduce the problem identified earlier on SUSE using go 1.10.5. Were there any env variables used during the build? What distro was it.

@hramrach
Copy link

@hramrach hramrach commented Oct 17, 2019

In openSUSE there is a package gcc-PIE which sets the gcc defaults options to PIC and PIE (since Leap 15.0 which is the oldest version still developed). They can still be overridden on the commandline or the package may opt to not install these defaults.

Not using the PIE options was what was done in openSUSE to build go and go packages on ppc64le.
I can look up the current state if there is some interest in debugging this further.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 20, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 20, 2019

@hramrach Please be polite when making comments, and bear in mind the Gopher Code of Conduct (https://golang.org/conduct). In particular, be patient and be charitable. Thanks.

@hramrach
Copy link

@hramrach hramrach commented Oct 20, 2019

What the R_PPC64_ADDR16_HA suggests is you generate direct references which are filled in as immediates in the code which are only 16bit and you need to generate indirect jumps or loads to be able to link non-trivial ppc binaries referencing code or data more than +-32kb away.

@jfkw
Copy link

@jfkw jfkw commented Oct 20, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

@ianlancetaylor Currently, the openSUSE ppc64le go package builds by not installing the gcc-PIE package. I will do some testing in openSUSE ppc64le using the GOFLAGS=-buildmode=pie used uniformly for building go, go tools and small programs written in go, and report back here in the coming weeks to correlate results with the observations from @laboger.

@hramrach
Copy link

@hramrach hramrach commented Oct 21, 2019

I suspect this is a bootstrap issue to some extent. You cannot build pie binaries if your stdlib is not PIC. You probably need to rebuild twice, once to build PIC stdlib injecting the PIC flag somewhere, and second time to build the PIE binaries.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach If I understand you correctly, then the go tool should consistently do the right thing. Using go build -buildmode=pie will build all Go code, including the standard library, as appropriate for PIE. Using go build without -buildmode=pie will build all Go code, including the standard library, as appropriate for ordinary executables.

@hramrach
Copy link

@hramrach hramrach commented Oct 21, 2019

that's not possible in the case when you are trying to build a pie compiler and all you have is non-PIE compiler with non-PIC stdlib. The new compiler will use the old stdlib which is not suitable. So you first need to build a compiler with PIC stdlib which does not default to PIE and then fully PIE compiler. Assuming the code generation works and correct relocations are generated.

@laboger
Copy link
Contributor Author

@laboger laboger commented Oct 21, 2019

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

Yes, if I use -x I can see it uses -buildmode=pie on the final link of the executable for hello.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach I'm sorry, I'm not sure what you mean. When you say "stdlib", what are you referring to? When you say "interpreter", do you mean the ELF interpreter, also known as the dynamic linker?

@hramrach
Copy link

@hramrach hramrach commented Oct 21, 2019

sorry, I meant compiler. Looks like interpreter is not standard part of go.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

@hramrach When you say "stdlib", what are you referring to?

@hramrach
Copy link

@hramrach hramrach commented Oct 21, 2019

whatever comes with the compiler that might be linked into binaries.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 21, 2019

Assuming you mean the Go compiler, then, as I explained at #28531 (comment), the Go standard library will be rebuilt, if necessary, with the options required to support the requested build mode. This happens automatically and does not require any user intervention. The results are cached to avoid excess rebuilding.

@laboger
Copy link
Contributor Author

@laboger laboger commented Oct 21, 2019

After I do a normal build, I can do this to make these binaries built as pie:

go install -buildmode=pie cmd/go
go install -buildmode=pie cmd/trace
go install -buildmode=pie cmd/pprof

Does this help?

@laboger
Copy link
Contributor Author

@laboger laboger commented Feb 27, 2020

I would like to find a solution for this in Go 1.15. There are several options but they all have build failures due to the cache. I would need some help from someone who knows more about how the build cache is supposed to work.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Feb 27, 2020

It's not clear to me what the build cache is doing wrong. Can you provide more details?

@laboger
Copy link
Contributor Author

@laboger laboger commented Feb 28, 2020

I tried various combinations to force binaries to pie but could get nothing to work. Back to original example above, if I try to force everything built for ppc64le to -buildmode=pie then I get the stale errors above. Since I'm not sure how the cache is supposed to work, I don't know why it is complaining. I thought I saw a comment somewhere that said the buildmode was not saved in the cache which could be related but I still don't understand how something would get into the cache at one point and then look stale later if the same build options were used throughout.

Assuming we could get it to work using pie as default, then there could be other issues. One would be how this default gets set, and the fact that we now check gcc's default link mode and change it from pie to no-pie all the time. That would have to change, and maybe that should be the indicator as to when to set Go's default buildmode to pie? If gcc wants pie as default then Go's buildmode should default to pie. Otherwise we provide an environment variable, maybe one used at build time to say this toolchain defaults to pie?

Another question is, does this mean everything defaults to pie, even binaries that would have been built as static? I believe static should still work fine on systems that expect pie as the default unless there is some other reason we don't want static in this environment.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Feb 28, 2020

The "stale dependency" errors shown in #28531 (comment) aren't about the build cache. Those are coming from cmd/dist, and mean that the package installed by go install is stale by comparison to the source files. When cmd/dist has finished building the tools, it runs a go list command to find all package for which the installed version is considered to be stale. See checkNotStale in cmd/dist/build.go. In your case this may have something to with the -gcflags and -ldflags that cmd/dist passes to go list.

@laboger
Copy link
Contributor Author

@laboger laboger commented Feb 28, 2020

I thought the original suggestion was to set the default buildmode to pie and codegenArg to shared in init.go. Doing that gives the output shown above.

Then I tried modifying lib.go to avoid the errors from loadInternal about runtime/cgo. That eliminates those errors but the rest of the error message is still there.

I did spend some time looking at checkNotStale but I don't understand why go list is complaining that these packages are stale because I don't know how it decides that and looking at the code didn't help to figure that out. Even if I look at the go list help documentation, it doesn't say what causes something to be stale.

Are you saying that the -gcflags and -ldflags passed to go list in this function should use flags for pie and shared instead of blank as the output shows?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 1, 2020

I'm sorry, I guess I don't understand what is going on.

This issue started as a problem using -buildmode=pie. Do you still have that problem?

Then there was the separate discussion of changing the toolchain to default to generating a PIE. I guess the suggestion of changing cmd/go/internal/work may have been wrong. What was the patch that you have tried? I don't it above; maybe I missed it.

@laboger
Copy link
Contributor Author

@laboger laboger commented Mar 2, 2020

I'm trying to build a toolchain that generates pie binaries by default, including the binaries that are generated during the build. Then the goal would be to have some environment variable or other switch to indicate that it should be done that way. Sorry for the confusion. This has been requested by more than one user but I personally do not have any specific issues with reproducible directions on how to cause errors when we don't default to pie on systems that expect pie. Based on previous test failures we saw with this error message when building for pie, it is probably related to mixing pic and non-pic code which we may or may not be able to fix. (i.e., if linking in a C library that was not built for pic).

I tried a change to cmd/go/internal/work to force buildmode=pie by default, but that causes the errors during the build about runtime/cgo and stale packages coming from checkNotStale in cmd/dist/build.go.

I don't understand why go -list would say the package or binary is stale if it just built it and therefore should be newer than the source. What else would cause it to be considered stale.

Here is an initial patch to set the default to pie and fix up the errors about runtime/cgo. I also added something to remove defaulting to no-pie to see if that made any difference but it doesn't.

golang-pie-simple.diff.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants