Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: unexpected trampoline when cross-compiling to ppc64le #45564

Closed
dbenoit17 opened this issue Apr 14, 2021 · 11 comments
Closed

cmd/link: unexpected trampoline when cross-compiling to ppc64le #45564

dbenoit17 opened this issue Apr 14, 2021 · 11 comments

Comments

@dbenoit17
Copy link

@dbenoit17 dbenoit17 commented Apr 14, 2021

What version of Go are you using (go version)?

$ go version
1.16.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOVERSION="go1.16.3"

What did you do?

Cross compile OpenShift's oc command from x86_64 to ppc64le.

git clone https://github.com/openshift/oc.git && pushd oc
make cross-build-linux-ppc64le

What did you expect to see?

Program builds.

What did you see instead?

Linker error at

ctxt.Errorf(s, "unexpected trampoline for shared or dynamic linking")

# github.com/openshift/oc/cmd/oc
github.com/openshift/oc/pkg/cli/observe.(*ObserveOptions).Run.func3: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.newlineTrailingWriter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.injectUserVars: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.processTemplateLocally: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).Complete.func2: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/process.processPrinter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/project.ProjectOptions.Run: unexpected trampoline for shared or dynamic linking
/root/go/pkg/tool/linux_amd64/link: too many errors

Additional Notes

  • This issue is not seen when compiling natively on ppc64le, only during cross-compilation from x86_64.
  • The issue was not seen prior to upgrading openshift/oc to use kubernetes 1.21.
  • The issue also occurs in Go versions 1.15.8 through 1.15.10, which are the only versions of 1.15 I've tested. However, it produces the error against different oc functions in Go 1.15 vs. Go 1.16.
  • Interestingly, 1.16 and 1.16.1 seem unaffected. The issue appears again in 1.16.2, and seems to be triggered by a9547ad#diff-aa02dd8eebb2420ba09bee48832b239fc5f612a07cb7b1709595511fa1d8d0ca. This commit might just uncover the issue rather than be it's direct cause, though, I'm not certain.
@mknyszek mknyszek changed the title Unexpected trampoline when cross-compiling to ppc64le cmd/link: unexpected trampoline when cross-compiling to ppc64le Apr 14, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 14, 2021

It is on my plan to rework trampoline insertion in the linker, that should address this. Hopefully I can get it in for 1.17.

@dbenoit17
Copy link
Author

@dbenoit17 dbenoit17 commented Apr 14, 2021

Thanks for the quick feedback!

I'm curious if there might be also a possibility for an interim patch to 1.16 in the meantime. One thing that does seem to get the linker past the issue is adjusting this condition to check for ctxt.IsExternal(), to closer match the earlier conditional here. I'm not sure if this is the correct thing to do though. I think it depends whether it's valid for the linker to arrive at line 706 with any of those configuration options when using internal linking.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 14, 2021

@dbenoit17 thanks for the comment. I'll take a careful look.

yselkowitz added a commit to multi-arch/openshift-installer that referenced this issue Apr 26, 2021
Recent versions of Go have had issues statically linking some ppc64le
binaries:

golang/go#45564
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Apr 26, 2021

@dbenoit17 sorry for the delay. As you said, I don't think we use internal linking for any of those configurations. I.e. IsExternal should always be true at that point.

Could you share how the "go" command or the linker is invoked in your build process, and what build mode are you building for? Thanks.

@pmur
Copy link
Contributor

@pmur pmur commented Apr 27, 2021

This seems to be caused by the combination of:
CGO_ENABLED=0 and something in openshift/installer using the plugin package causing canUsePlugins to be set, which causes (*Link).DynlinkingGo() to evaulate true, and causes the panic noted above.

Given that plugins are effectively disabled in the above case, shouldn't canUsePlugins also be set to false when linking?

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 27, 2021

Change https://golang.org/cl/314449 mentions this issue: cmd/link: disable plugin support if cgo is explicitly disabled

@dbenoit17
Copy link
Author

@dbenoit17 dbenoit17 commented Apr 27, 2021

To hit the issue, go build is called against openshift/oc commit 90af8a5d7a1ae91e3e675607c4f76a403e6ec817 by the Makefile using CGO_ENABLED=0, with the invocation:

go build -mod=vendor -tags 'include_gcs include_oss containers_image_openpgp' -ldflags "-X github.com/openshift/oc/pkg/version.versionFromGit="v4.2.0-alpha.0-1048-g90af8a5" -X github.com/openshift/oc/pkg/version.commitFromGit="90af8a5d7" -X github.com/openshift/oc/pkg/version.gitTreeState="clean" -X github.com/openshift/oc/pkg/version.buildDate="<date>" -X k8s.io/component-base/version.gitMajor="1" -X k8s.io/component-base/version.gitMinor="21" -X k8s.io/component-base/version.gitVersion="v0.21.0-beta.1" -X k8s.io/component-base/version.gitCommit="90af8a5d7" -X k8s.io/component-base/version.buildDate="<date>" -X k8s.io/component-base/version.gitTreeState="clean" -X k8s.io/client-go/pkg/version.gitVersion="v4.2.0-alpha.0-1048-g90af8a5" -X k8s.io/client-go/pkg/version.gitCommit="90af8a5d7" -X k8s.io/client-go/pkg/version.buildDate="<date>" -X k8s.io/client-go/pkg/version.gitTreeState="clean"" -o '_output/bin/linux_ppc64le/oc' github.com/openshift/oc/cmd/oc

I think this is using using buildmode=default since there is no buildmode specified.

As a follow-up, I've had a chance to test https://golang.org/cl/314449 and can confirm it resolves the issue.

@gopherbot gopherbot closed this in 983dea9 Apr 27, 2021
@laboger
Copy link
Contributor

@laboger laboger commented Apr 27, 2021

I think this is using using buildmode=default since there is no buildmode specified.

It was originally thought that the messages about dynamic linking were due to a buildmode setting, but that turned out not to be the case. Code that uses plugins is usually compiled as PIC (i.e., like shared or dynamic linking) but if CGO_ENABLED=0 then plugins are disabled and then not compiled as PIC so that led to confusion in the linker and the errors.

@pmur
Copy link
Contributor

@pmur pmur commented Apr 28, 2021

@gopherbot please consider a backport to 1.16. This seems to a be a regression from go 1.16.1 to 1.16.2.

I don't think a backport to earlier versions is necessary since the issue likely affects all 1.15 releases. It might be possible to workaround by not disabling cgo, or using external linking.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 28, 2021

Backport issue(s) opened: #45831 (for 1.15), #45832 (for 1.16).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants