New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: ppc64 (big endian) cgo errors #13192

Open
laboger opened this Issue Nov 9, 2015 · 30 comments

Comments

Projects
None yet
@laboger
Contributor

laboger commented Nov 9, 2015

I've built golang from master with the patches from issue 11184 to get external linking to work with ppc64le. That seems to work well on ppc64le with external linking and cgo.

In cmd/dist/build.go, linux/ppc64le is in the cgoEnabled map but linux/ppc64 is not. So the build of golang with the latest patches on ppc64 does not quite work with cgo.

If I build the golang toolchain on ppc64 with CGO_ENABLED=1, I first get this error:

cannot use dynamic imports with -d flag

I made a change to cmd/link/internal/ppc64/obj.go to get rid of this message, but then hit this error:
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
......
runtime/cgo(.opd): unexpected relocation type 307
.......
and then too many errors

I can see that relocation type 51 is R_PPC64_TOC and that is not handled by the code. I tried adding those defines but not sure what should be generated for this relocation type. If there are suggestions on what to do I can try them out.

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 9, 2015

I've been doing this on a RHEL7.2 system.

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Nov 9, 2015

Those errors suggest that you're not actually linking externally. Try
adding -ldflags=-linkmode=external to your go tool invocation.

I'm pretty sure that things still won't work though.

On 10 November 2015 at 09:25, laboger notifications@github.com wrote:

I've been doing this on a RHEL7.2 system.


Reply to this email directly or view it on GitHub
#13192 (comment).

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 9, 2015

These errors happen during the build of golang on ppc64 if I build with CGO_ENABLED=1.
They don't happen if CGO_ENABLED is not set.

Under ##### Building packages and commands for linux/ppc64.
.....
cmd/pprof
net/rpc/jsonrpc

cmd/trace

/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/net.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
/home/boger/golang/gitsrc/latest/go/pkg/linux_ppc64/runtime/cgo.a(_all.o): unknown relocation type 51; compiled without -fpic?
runtime/cgo(.opd): unexpected relocation type 307
runtime/cgo(.opd): unexpected relocation type 307
/home/boger/golang/gitsrc/latest/go/pkg/tool/linux_ppc64/link: too many errors

@ianlancetaylor ianlancetaylor added this to the Go1.6 milestone Nov 9, 2015

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Nov 9, 2015

@laboger so you're saying that the patches from issue 11184 break building go on ppc64? that's bad, I'll make sure to fix that. I'm not promising to make external linking work on ppc64 though (which is what I thought this was about).

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 9, 2015

I'm sorry that I wasn't clear. I'm not saying your patches for 11184 break the build of ppc64. The above errors occur if I build on ppc64 and set CGO_ENABLED=1. The errors are the same or similar if I try to do the same build without your patches on ppc64.

You are correct, the point of this issue is to document the errors that I've seen with external linking on ppc64. I used your patches in my testing just because they are the latest changes I'm aware of for linking on Power. I understand you are not promising to make it work for ppc64, but I'm trying to understand how much is missing from external linking for ppc64.

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 9, 2015

This issue should not affect the merging of your patches from 11184. They work great and I know of no problem with them. They are for ppc64le, this issue is ppc64.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Nov 9, 2015

Just to make sure we're all on the same page, these errors are from internal linking, which is what the toolchain does when building the tools. Presumably something has changed causing the C compiler to generate R_PPC64_TOC relocs. The code in cmd/link/internal/ppc64/asm.go needs to handle R_POWER_TOC, by resolving it to the value of symtoc. The code in cmd/link/internal/ld/ldelf.go needs to recognize R_PPC64_TOC, setting the size to 8.

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Nov 10, 2015

FWIW, I'm not sure that cgo has ever worked on ppc64 BE (ping @aclements @minux), so it might not be C compiler changes to blame.

@minux

This comment has been minimized.

Member

minux commented Nov 10, 2015

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 10, 2015

I have some old patches that Minux gave me back on May to make external linking work. There are only a few ppc64 BE specific changes that aren't in Michael's. I will try adding them and see if that helps. Perhaps something in his patches prevents the generation of R_PPC64_TOC.

My RHEL 7.2 ppc64 BE machine defaults to gcc 4.8.5.

@laboger

This comment has been minimized.

Contributor

laboger commented Nov 11, 2015

Here are the CLs that I am aware of related to ppc64 BE linking which match the patches I had previously used from Minux:
https://go-review.googlesource.com/#/c/9677
change to obj.go no longer needed due to MHDs change
https://go-review.googlesource.com/#/c/9676
change to tls_ppc64x.s no longer needed due to MHDs change in TLS handling
I applied the other changes from this CL.
https://go-review.googlesource.com/#/c/9673
change to obj.go no longer needed

The file asm.go from 9677 and 9673 is a question because I see there are conflicting changes now with what's upstream.

I tried to add support to handle R_PPC64_TOC based on Ian's comments above but was unsure what to put in adddynrel for the R_PPC64_TOC case.

And then after making those changes, I see errors like this:

TOC-relative relocation in object without .TOC.

I don't see where a .TOC. object would get generated so it would be found when calling Linkrlookup.

I'm not sure what the next step is to make this work. I agree with previous comments that when CGO_ENABLED=1 on ppc64 BE this has probably never worked. I can see that gcc generates .opd sections with the R_PPC64_TOC relocation type and that happens even with older versions of gcc. I found this .opd section in some of the _all.o files that are generated during the golang build that are generating the error message about the missing relocation type.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 5, 2015

We're not going to get to this for Go 1.6.

@rsc rsc modified the milestones: Unplanned, Go1.6 Dec 5, 2015

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Dec 6, 2015

This bug is assigned to me but I'm not in a position to work on it. Can someone unassign me? (I don't think I have permissions to do that!)

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Dec 6, 2015

Unassigned.

@laboger

This comment has been minimized.

Contributor

laboger commented Dec 7, 2015

Can I get information on what is left to be done to get this to work?

On 12/04/2015 11:55 PM, Russ Cox wrote:

We're not going to get to this for Go 1.6.


Reply to this email directly or view it on GitHub
#13192 (comment).

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 7, 2015

@laboger, my understanding is that the patches Minux posted a while back do work (I think you've said that), but they no longer apply cleanly, and Minux has not had time to update them to the latest Go tree. (Minux is a volunteer; his day job is being a grad student.)

If you or anyone else would like to take over those patches, get them to apply to the tree, and send them in for review, we can get them in for Go 1.6.

More generally, my understanding was that ppc64 (big-endian) was not as important as ppc64le, and that ppc64le support is committed and working. If that's wrong please let me know. Thanks.

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Dec 7, 2015

I don't know if ppc64 is less important than ppc64le in general, but certainly to me (and Canonical) it is, as there is no Ubuntu port to ppc64 (and I don't have access to ppc64 hw, so even if I wanted to work on it, in practice I can't).

I don't think there is an enormous amount to be done.

@laboger

This comment has been minimized.

Contributor

laboger commented Dec 7, 2015

Yes your understanding is true -- ppc64 for big endian is not as
important at this time. It was very important
to get ppc64le working in Go 1.6 with external linking and that seems to
be working well and that is a very good thing.

This question is mainly for my understanding, in case the importance of
BE changes and I'm asked what it
would take to make it work. I wasn't sure if it was thought that Minux'
previous patches contained most of the needed function
or if there were some known pieces missing. I might have thought at one
time they worked OK on BE but now I'm not sure on that
because I probably didn't understand the variations that needed testing.

On 12/07/2015 10:48 AM, Russ Cox wrote:

@laboger https://github.com/laboger, my understanding is that the
patches Minux posted a while back do work (I think you've said that),
but they no longer apply cleanly, and Minux has not had time to update
them to the latest Go tree. (Minux is a volunteer; his day job is
being a grad student.)

If you or anyone else would like to take over those patches, get them
to apply to the tree, and send them in for review, we can get them in
for Go 1.6.

More generally, my understanding was that ppc64 (big-endian) was not
as important as ppc64le, and that ppc64le support is committed and
working. If that's wrong please let me know. Thanks.


Reply to this email directly or view it on GitHub
#13192 (comment).

Can I get information on what is left to be done to get this to work?

On 12/04/2015 11:55 PM, Russ Cox wrote:

We're not going to get to this for Go 1.6.


Reply to this email directly or view it on GitHub
#13192 (comment).

@aleek

This comment has been minimized.

aleek commented Feb 17, 2016

I believe that I've found a way to cross compiler binaries for ppc64be. I'm lookking forward to some feedback.
http://dutkowski.me/index.php/2016/02/17/crosscompiling-go-with-c-library-on-powerpc64/

regards
Alex

@tdolby

This comment has been minimized.

tdolby commented May 18, 2016

@laboger I've got to the same point as you have (followed Ian's instructions and now the build can't find .TOC. entries) and it looks like the ELF v2 ABI isn't implemented on ppc64be in any of the gcc versions I've tried (4.4, 4.8, 5) , which is why we don't see .TOC. entries being created.

This can verified by running strings on a generated .o file created using gcc on big- and little-endian and comparing the output. I've tried specifying -mabi=elfv2 on the gcc command line to force it on big-endian, but it's unable to find gnu/stubs-64-v2.h and the compile fails. I've also tried to hack the cgo code to use .toc (lower case), but that's not worked either.

Looks like a rebuild of gcc with ABI v2 might be needed to get things going, or cross-compiling as @aleek has done.

@laboger

This comment has been minimized.

Contributor

laboger commented May 19, 2016

ppc64 BE uses ABI v1 and ppc64 LE uses ABI v2. You can't build gcc on a ppc64 BE machine with ABI v2, because everything else on a ppc64 BE machine expects ABI v1 (dynamic linker, assembler, shared libraries, etc.) The golang for ppc64 BE must generate the linking environment that is consistent with ABI v1 for it to link correctly and run there.

I did some further investigation on this back when I opened the issue which I didn't add to this issue. There is a lot missing in golang for GOARCH=ppc64 with respect to external linking on ABI v1. The call stubs are not correct, the PLT is not set up correctly, plus probably some other things, in addition to the defining of the .TOC. symbol. @tdolby

@houstar

This comment has been minimized.

houstar commented Jun 29, 2016

@laboger How much work If we're enable golang with cgo on ppc64 BE ? I'm curious about it because we're interesting about docker on Power7 Machine that we purchased before. Thanks.

buildroot-auto-update pushed a commit to buildroot/buildroot that referenced this issue Nov 11, 2016

go: remove powerpc64 big-endian from supported architectures
cgo currently doesn't properly support powerpc64 big-endian, as noted
in golang/go#13192, and indeed, we have a
large number of build failures of Go packages on this
architecture. This commit therefore disables Go on PowerPC64
big-endian (PowerPC64 little-endian is fine).

Fixes:

  http://autobuild.buildroot.net/results/a6e9bac0a735f48d0ba0af081aeac4ed9fdfaca7/
  (flannel)

  http://autobuild.buildroot.net/results/230f52bc35f437836c7a76d4b58ef454635ee0d3/
  (docker-containerd)

  http://autobuild.buildroot.net/results/77c31d6e8f5efe3e024e27a160cf5d1d1952719e/
  (runc)

  http://autobuild.buildroot.net/results/a87b07417ea8bd81ffe27e5661b4359ddc0149ab/
  (docker-engine)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
@markos

This comment has been minimized.

markos commented Jul 19, 2017

@laboger @minux @aclements hi and sorry for tagging all you guys, I've started implementing the ppc64(be) bits on internal linker and am hitting some roadblocks. It's harder than I expected initially, but I am prepared to do the work, just need some more specific directions (yes I know that the TOC entries are wrong etc), but apart from the fact that ABIv1 is more complicated than I thought. I think I saw minux in an older forum claiming to have ppc64be working but the links to the patches were broken (they pointed to the older code.google site).
What I've done so far is enabled cgo support in the code, added ppc64 wherever was needed for shared builds/linking, added the R_PPC64_TOC case in adddynrel() in cmd/link/internal/ppc64/asm.go, but symtoc (obviously) fails to find the TOC entry (r.Sym is nil).
Then I added the .TOC base entry in the src/cmd/link/internal/ld/elf.go (using DT_PPC64_TOC = 0x8000). So TOC base exists, and I need to add the TOC+symbol offset to the Symbols table, for each symbol. However when I do that (in genplt() allowing when r.Type == 256+ R_PPC64_TOC), I'm getting "unexpected R_PPC64_REL24 for dyn import" in adddynrel(). I guess I'm adding the symbol in the wrong place.

I'd appreciate any help.

@jrtc27

This comment has been minimized.

Contributor

jrtc27 commented Jul 19, 2017

@markos what symbol are you referencing? Is it really local or external? Calls to external functions are also done with a standard bl, and it's up to the linker to insert a trampoline to load the actual callee's address and TOC from the function descriptor, the address of which is loaded from the caller's TOC. My guess is that's not being done properly; see https://uclibc.org/docs/psABI-ppc64.pdf, section 3.5.11, for the details.

@laboger

This comment has been minimized.

Contributor

laboger commented Jul 19, 2017

Have you considered using gccgo? That can be built for ppc64 big endian, configured for several instruction sets, and the linking works. When it is built, it has its own go tool, so once that is in your path it should work for the most part like golang. In some cases, you might need to make some changes to your build scripts (e.g. if you want special options) but that is much less painful than the changes that would be needed in golang to get linking to work on ppc64 big endian.

BWhitten added a commit to LairdCP/wb-buildroot that referenced this issue Dec 14, 2017

go: remove powerpc64 big-endian from supported architectures
cgo currently doesn't properly support powerpc64 big-endian, as noted
in golang/go#13192, and indeed, we have a
large number of build failures of Go packages on this
architecture. This commit therefore disables Go on PowerPC64
big-endian (PowerPC64 little-endian is fine).

Fixes:

  http://autobuild.buildroot.net/results/a6e9bac0a735f48d0ba0af081aeac4ed9fdfaca7/
  (flannel)

  http://autobuild.buildroot.net/results/230f52bc35f437836c7a76d4b58ef454635ee0d3/
  (docker-containerd)

  http://autobuild.buildroot.net/results/77c31d6e8f5efe3e024e27a160cf5d1d1952719e/
  (runc)

  http://autobuild.buildroot.net/results/a87b07417ea8bd81ffe27e5661b4359ddc0149ab/
  (docker-engine)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
@zeldin

This comment has been minimized.

zeldin commented May 12, 2018

@markos Are you still working on this? I could try helping out... Do you have a branch on github with your work?

SuperQ added a commit to prometheus/promu that referenced this issue Aug 13, 2018

Add ppc64 build workaround
Due to golang/go#13192, we need to pass in
a tweak to the ldflags.

Signed-off-by: Ben Kochie <superq@gmail.com>

SuperQ added a commit to prometheus/promu that referenced this issue Aug 13, 2018

Add ppc64 build workaround
Due to golang/go#13192, we need to pass in
a tweak to the ldflags.

Signed-off-by: Ben Kochie <superq@gmail.com>

Risca added a commit to Risca/CHIP-buildroot that referenced this issue Sep 30, 2018

go: remove powerpc64 big-endian from supported architectures
cgo currently doesn't properly support powerpc64 big-endian, as noted
in golang/go#13192, and indeed, we have a
large number of build failures of Go packages on this
architecture. This commit therefore disables Go on PowerPC64
big-endian (PowerPC64 little-endian is fine).

Fixes:

  http://autobuild.buildroot.net/results/a6e9bac0a735f48d0ba0af081aeac4ed9fdfaca7/
  (flannel)

  http://autobuild.buildroot.net/results/230f52bc35f437836c7a76d4b58ef454635ee0d3/
  (docker-containerd)

  http://autobuild.buildroot.net/results/77c31d6e8f5efe3e024e27a160cf5d1d1952719e/
  (runc)

  http://autobuild.buildroot.net/results/a87b07417ea8bd81ffe27e5661b4359ddc0149ab/
  (docker-engine)

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
@zte-majiang

This comment has been minimized.

Contributor

zte-majiang commented Oct 25, 2018

Hi all, we have implemented ppc64be cgo on golang1.10.3. With it (and a custom GNU toolchain) , We have successfully built docker_17.03.2-ce, and done some basic tests (on our ppc64 e6500 cores). Everything looks fine. I'll try to upstream these patches later (after we get permissions from our boss ...).
@laboger @minux @aclements @markos @zeldin

@laboger

This comment has been minimized.

Contributor

laboger commented Oct 25, 2018

@majiang31312

This comment has been minimized.

majiang31312 commented Oct 26, 2018

Just curious, why do you need a custom GNU toolchain?

The GNU toolchain used in ppc64 cgo tests is maintained internally (also by our team) , that is why I call it "custom".
In theory, a standard version should also be OK. But we have not test that case yet.

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 2, 2018

cgo: add initial support for ppc64
The patch adds initial cgo support for ppc64.
We should be able to build docker after this get applied.

Fixes golang#13192
@gopherbot

This comment has been minimized.

gopherbot commented Nov 2, 2018

Change https://golang.org/cl/146898 mentions this issue: cgo: add initial support for ppc64

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 2, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates golang#13192

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 2, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates golang#13192

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 13, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates golang#13192

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 13, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates golang#13192

zte-majiang added a commit to zte-majiang/go that referenced this issue Nov 16, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates golang#13192

gopherbot pushed a commit that referenced this issue Nov 20, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates #13192

Change-Id: I5378d3518fac52d6bd4c97828884c1b382b7ace5
GitHub-Last-Rev: 210b7bc
GitHub-Pull-Request: #28546
Reviewed-on: https://go-review.googlesource.com/c/146898
Reviewed-by: Jiang Ma <ma.jiang@zte.com.cn>
Reviewed-by: Clément Chigot <clement.chigot@atos.net>
Reviewed-by: Lynn Boger <laboger@linux.vnet.ibm.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

bradfitz pushed a commit that referenced this issue Nov 21, 2018

cmd/link, runtime: add initial cgo support for ppc64
We should be able to build docker after this get applied.

Updates #13192

Change-Id: I5378d3518fac52d6bd4c97828884c1b382b7ace5
GitHub-Last-Rev: 210b7bc
GitHub-Pull-Request: #28546
Reviewed-on: https://go-review.googlesource.com/c/146898
Reviewed-by: Jiang Ma <ma.jiang@zte.com.cn>
Reviewed-by: Clément Chigot <clement.chigot@atos.net>
Reviewed-by: Lynn Boger <laboger@linux.vnet.ibm.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment