Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: don't prefer external linking when using bazelbuild/rules_go #60865

Closed
zecke opened this issue Jun 19, 2023 · 5 comments
Closed

cmd/go: don't prefer external linking when using bazelbuild/rules_go #60865

zecke opened this issue Jun 19, 2023 · 5 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@zecke
Copy link

zecke commented Jun 19, 2023

What version of Go are you using (go version)?

$ go version
1.20.3

Does this issue reproduce with the latest release?

Yes. src/cmd/go/internal/work/security.go appears unchanged

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

What did you do?

I am using bazel + rules_go to compile an application that requires cgo (e.g., uses os/user) and have upgraded from Go 1.19 to 1.20. The build set-up is described in bazelbuild/rules_go#3590.

What did you expect to see?

In Go 1.19 the Go linker is linking the final binary. I expected this to continue.

What did you see instead?

Go 1.20 is actually invoking the external linker.

Looking at the linker output I notice:

external linking is forced by: some packages could not be built to support internal linking ([os/user runtime/cgo])
external linking prefer list is [os/user runtime/cgo]

Inspecting the os/user.a generated it contains a preferlinkext entry. This seems to be the case because rules_go is passing -fdebug-prefix-map=... as CGO_CFLAGS. I would like to allow list this flag to src/cmd/go/internal/work/security.go

zecke pushed a commit to zecke/go that referenced this issue Jun 19, 2023
The -fdebug-prefix-map= option is passed by bazelbuild/rules_go
when building cgo code. This option impacts the generation of the
debug symbols but has no effect on linking. Add it to the allow-list
to not prefer external linking.

Fixes: golang#60865
@gopherbot
Copy link

Change https://go.dev/cl/504335 mentions this issue: cmd/go: Don't prefer external linking when using bazel

@bcmills bcmills added GoCommand cmd/go compiler/runtime Issues related to the Go compiler and/or runtime. labels Jun 20, 2023
@bcmills bcmills added this to the Backlog milestone Jun 20, 2023
@bcmills
Copy link
Member

bcmills commented Jun 20, 2023

In Go 1.19 the Go linker is linking the final binary. I expected this to continue.

Note that rules_go could force internal linking by setting GO_EXTLINK_ENABLED=0 explicitly in the environment (see https://pkg.go.dev/cmd/go#hdr-Environment_variables), or passing -ldflags=all=-linkmode=internal as an argument.

Since Bazel presumably knows whether it expects a C linker to be invoked, it should probably use one of those mechanisms explicitly to avoid making fragile assumptions about which flags do or do not trigger external linking by default.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 20, 2023
@mknyszek mknyszek changed the title cmd/go: Don't prefer external linking when using bazelbuild/rules_go cmd/go: don't prefer external linking when using bazelbuild/rules_go Jun 28, 2023
@cherrymui
Copy link
Member

Allowing -fdebug-prefix-map= for internal linking seems reasonable to me. Thanks.

As @bcmills mentioned, explicitly passing -ldflags=-linkmode=internal is probably a good idea if this is what Bazel wants.

@cherrymui cherrymui assigned zecke and unassigned cherrymui Jun 28, 2023
@zecke
Copy link
Author

zecke commented Jul 5, 2023

In Go 1.19 the Go linker is linking the final binary. I expected this to continue.

Note that rules_go could force internal linking by setting GO_EXTLINK_ENABLED=0 explicitly in the environment (see https://pkg.go.dev/cmd/go#hdr-Environment_variables), or passing -ldflags=all=-linkmode=internal as an argument.

Since Bazel presumably knows whether it expects a C linker to be invoked, it should probably use one of those mechanisms explicitly to avoid making fragile assumptions about which flags do or do not trigger external linking by default.

My understanding is that the go command is looking at each input, cgo, msan, asan, the GOOS and GOARCH to resolve linkmode=auto (determineLinkMode) and then looks whether it can proceed (extNeeded).

It seems that unconditionally passing flags to force internal linking, certain architectures and OSes will fail to link. It seems undesirable and close to impossible (bazel does not inspect the .o files) to resolve LinkAuto in rules_go.

Why does this behavior change matter? I have orders of magnitude more confidence in Go doing the right thing than the less frequently updated random C toolchain.

@cherrymui
Copy link
Member

Why does this behavior change matter? I have orders of magnitude more confidence in Go doing the right thing than the less frequently updated random C toolchain.

This sounds like you really want internal linking on certain condition. If this is a requirement for bazel, it still makes sense to make that explicitly. (This is not a requirement for the Go toolchain, at least not a hard requirement. So it an still change in the future.)

You don't need to inspect object files. Basically, any (non-std) cgo, msan, or asan means external linking, as well as a small number of special GOOS/GOARCH combinations where internal linking is not supported/implemented. This is the default behavior for the linker.

zecke pushed a commit to zecke/go that referenced this issue Jul 9, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Last-Rev: 4127df6
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Jul 9, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Last-Rev: 4127df6
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Jul 9, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

As a side effect, it also allows these flags to appear
in #cgo directives in source code. We don't know of any cases
where that is actually useful, but it appears to be harmless
and simplifies the implementation of the internal linking change.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Last-Rev: 4127df6
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Jul 9, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

As a side effect, it also allows these flags to appear
in #cgo directives in source code. We don't know of any cases
where that is actually useful, but it appears to be harmless
and simplifies the implementation of the internal linking change.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Jul 9, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

As a side effect, it also allows these flags to appear
in #cgo directives in source code. We don't know of any cases
where that is actually useful, but it appears to be harmless
and simplifies the implementation of the internal linking change.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Sep 23, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

As a side effect, it also allows these flags to appear
in #cgo directives in source code. We don't know of any cases
where that is actually useful, but it appears to be harmless
and simplifies the implementation of the internal linking change.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Pull-Request: golang#60868
zecke pushed a commit to zecke/go that referenced this issue Sep 23, 2023
In CL 475375 the Go command started to generate the "preferlinkext"
token file for "strange/dangerous" compiler flags. This serves as a hint
to the Go linker whether to call the external linker or not.

Permit compiler flags used by bazel and bazelbuild/rules_go during
compilation of cgo code to not prefer external linking. This restores
the behavior of previous versions of Go.

As a side effect, it also allows these flags to appear
in #cgo directives in source code. We don't know of any cases
where that is actually useful, but it appears to be harmless
and simplifies the implementation of the internal linking change.

Fixes golang#60865

Change-Id: I176a6a2a2cf36293dd9aed24be928f98fa2fb6d9
GitHub-Pull-Request: golang#60868
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants