Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: coverage analysis fail when using CGO & LLVM/clang 12 #58848

Open
leitzler opened this issue Mar 3, 2023 · 17 comments
Open

cmd/link: coverage analysis fail when using CGO & LLVM/clang 12 #58848

leitzler opened this issue Mar 3, 2023 · 17 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@leitzler
Copy link
Contributor

leitzler commented Mar 3, 2023

What version of Go are you using (go version)?

$ go version
go version devel go1.21-14015be5bb Fri Mar 3 01:33:00 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes, it reproduce with 1.19, 1.20 and gotip.

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

I'm running coverage analysis via Bazel, with CGO enabled, and LLVM 12 configured as llvm toolchain.
rules_go is the latest release, v0.38.1.

What did you expect to see?

Coverage analysis work as it did in Go 1.18.

What did you see instead?

Loadelf reporting symbols:

loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/crypto/internal/boring.a(_x006.o): 1076397: sym#7 (__covrec_33995B8EE71787FCu): ignoring symbol in section 13 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/crypto/internal/boring.a(_x007.o): 1076402: sym#7 (__covrec_620A963398FC4C69): ignoring symbol in section 9 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/crypto/internal/boring.a(_x008.o): 1076407: sym#7 (__covrec_10624AC03CC26411u): ignoring symbol in section 17 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/crypto/internal/boring.a(_x009.o): 1076412: sym#7 (__covrec_17B0C410E83682Au): ignoring symbol in section 37 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x001.o): 1076417: sym#7 (__covrec_93B800E74F424049u): ignoring symbol in section 5 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x002.o): 1076420: sym#4 (__covrec_1A8C6975D1B2592B): ignoring symbol in section 6 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x003.o): 1076425: sym#7 (__covrec_1611CD51FA32BE02u): ignoring symbol in section 5 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x004.o): 1076428: sym#4 (__covrec_4FDDED4762B8A734): ignoring symbol in section 6 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x005.o): 1076433: sym#7 (__covrec_5C060E46219FD9FCu): ignoring symbol in section 5 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/net.a(_x006.o): 1076438: sym#7 (__covrec_451659BD05925CB2): ignoring symbol in section 9 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x002.o): 1076442: sym#4 (__covrec_3BF569B9C342ECE1): ignoring symbol in section 4 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x003.o): 1076447: sym#7 (__covrec_E16FC1323EFFEF22u): ignoring symbol in section 5 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x004.o): 1076455: sym#9 (__covrec_2296CDEC6E5368C6u): ignoring symbol in section 6 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x005.o): 1076467: sym#13 (__covrec_2B7EBFBF5390F0FDu): ignoring symbol in section 11 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x006.o): 1076479: sym#13 (__covrec_68E3D53E6EE769C6u): ignoring symbol in section 11 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x007.o): 1076484: sym#7 (__covrec_7DEE5C08E39BE2EBu): ignoring symbol in section 7 (type 1)
loadelf: bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_/pkg/linux_amd64/runtime/cgo.a(_x008.o): 1076489: sym#7 (__covrec_55E440AE900CFC9u): ignoring symbol in section 5 (type 1)

It looks like the coverage generation using generated symbols named __covrec_* was added in LLVM 11 so I'd expect the same to happen on all versions after 11. This hasn't been tested.

If I disable CGO the coverage works.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 3, 2023
@gopherbot
Copy link

Change https://go.dev/cl/473115 mentions this issue: cmd/link: don't warn about llvm generated __covrec_ symbols

@leitzler
Copy link
Contributor Author

leitzler commented Mar 3, 2023

@thanm thanm added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 3, 2023
@thanm
Copy link
Contributor

thanm commented Mar 3, 2023

Thanks for the report.

It would be helpful if you could post the C flags that are being used for this build (e.g. what's being passed to the C compiler when the various CGO C files are compiled).

Also, I am curious as to what happens if you switch to external linking (e.g. "-ldflags=-linkmode=external"), whether that also resolve the issue.

@leitzler
Copy link
Contributor Author

leitzler commented Mar 3, 2023

Thank you for the quick reply!

    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -fno-omit-frame-pointer -Wthread-safety -Wself-assign --target=x86_64-linux-gnu -fPIC -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -fdebug-prefix-map=external/llvm_toolchain/=external/llvm_toolchain/ --sysroot=external/foo_sysroot_k8s -fprofile-instr-generate -fcoverage-mapping' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/foo_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/foo_sysroot_k8s -fprofile-instr-generate' \
    GOARCH=amd64 \
    GOOS=linux \

I belive that -fprofile-instr-generate -fcoverage-mapping is the ones that creates the coverage symbols.

I'll see if I can convince bazel to run with -linkmode=external :)

@leitzler
Copy link
Contributor Author

leitzler commented Mar 4, 2023

It looks like passing -linkmode=external doesn't do any difference, still fails.
Would it make sense to omit the __covrec_ symbols in the elf loader as per my CL above?

@leitzler
Copy link
Contributor Author

leitzler commented Mar 4, 2023

@gopherbot, please remove label WaitingForInfo

@gopherbot gopherbot removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 4, 2023
@thanm
Copy link
Contributor

thanm commented Mar 6, 2023

Thanks. The failure with -linkmode=external is surprising -- would you mind posting the error output for that?

@leitzler
Copy link
Contributor Author

leitzler commented Mar 6, 2023

Yeah, that was a bit surprising to me as well. I get the same error output from loadelf so I supposed it tries to load the elf before invoking the external linker?

@leitzler
Copy link
Contributor Author

leitzler commented Mar 6, 2023

The rules_go GoLink action step w/o -linkmode=external:

  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_ \
    GOROOT_FINAL=GOROOT \
    PATH=external/llvm_toolchain/bin:/bin:/usr/bin \
  bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/k8-fastbuild/bin/foo/bar/service/common/go_default_test_/go_default_test-0.params' -- -extld external/llvm_toolchain/bin/clang -s -w -X 'github.com/bazelbuild/rules_go/go/tools/bzltestutil.RunDir=foo/bar/service/common' '-buildid=redacted' -extldflags '-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/my_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/my_sysroot_k8s -fprofile-instr-generate --coverage')

And with -linkmode=external it is slightly different:

exec env - \
    CC=external/llvm_toolchain/bin/clang \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=bazel-out/k8-fastbuild-ST-b33d65c724e6/bin/external/io_bazel_rules_go/stdlib_ \
    GOROOT_FINAL=GOROOT \
    PATH=external/llvm_toolchain/bin:/bin:/usr/bin \
  bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/k8-fastbuild/bin/foo/bar/service/service.bin-0.params' -- -extld external/llvm_toolchain/bin/clang '-buildid=redacted' -extldflags '-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -Lexternal/llvm_toolchain//lib -l:libc++.a -l:libc++abi.a -l:libunwind.a -rtlib=compiler-rt -lpthread -ldl -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-z,relro,-z,now -Wl,--unresolved-symbols=ignore-in-object-files -Lexternal/my_sysroot_k8s/usr/lib/x86_64-linux-gnu --sysroot=external/my_sysroot_k8s -fprofile-instr-generate --coverage')

@thanm
Copy link
Contributor

thanm commented Mar 6, 2023

Hmm, based on what I am seeing I think the "ldflags=-linkmode=external" is maybe not making it through to the Go command... the reason I say this is that the "loadelf" routines in question (which are generating the errors) are only invoked during internal linking.

Here's a toy program that uses the os/user package:

package main

import "os/user"

func main() {
	if g, err := user.LookupGroup("mumble"); err != nil {
		println("that was susprising", g != nil)
	}
}

and here is a script I can use to compile it. Note the use of -fprofile-instr-generate -fcoverage-mapping etc.

#!/bin/sh
set -x
rm -f toy.exe
CC=clang-14 \
CGO_CFLAGS="-fstack-protector -fno-omit-frame-pointer -Wthread-safety -Wself-assign --target=x86_64-linux-gnu -fPIC -no-canonical-prefixes -Wno-builtin-macro-redefined -fprofile-instr-generate -fcoverage-mapping" \
  CGO_ENABLED=1 \
  CGO_LDFLAGS="-lm -no-canonical-prefixes --target=x86_64-linux-gnu -fuse-ld=lld -rtlib=compiler-rt -lpthread -ldl -Wl,--unresolved-symbols=ignore-in-object-files -fprofile-instr-generate" \
    GOARCH=amd64 \
    GOOS=linux \
    go build -o toy.exe $* toy.go

If I do a build with "-ldflags=-linkmode=external" this works fine (external linker can handle the sections generated by the coverage flags), whereas if I build with "-ldflags=-linkmode=internal" I get the same sorts of errors that you are seeing with your bazel build. Eg

loadelf: $WORK/b001/_pkg_.a(_x002.o): 53157: sym#5 (__profc_sayit): ignoring symbol in section 14 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x002.o): 53161: sym#4 (__covrec_3BF569B9C342ECE1): ignoring symbol in section 4 (type 1)
loadelf: $WORK/b003/_pkg_.a(_x003.o): 53166: sym#4 (__profc__cgo_release_context): ignoring symbol in section 7 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x004.o): 53174: sym#6 (__profc_fatalf): ignoring symbol in section 8 (type 0)
loadelf: $WORK/b003/_pkg_.a(_x005.o): 53196: sym#10 (__profc_x_cgo_sys_thread_create): ignoring symbol in section 19 (type 0)
...

See also related issues #58619 and #58620, similar things are happening there.

The best fix for all these problems is to make sure that we wind up using the external linker, as opposed to continuing to use the internal linker and teaching it to deal with the various quirks in host objects.

I am working on a CL 471055 that will help solve these problems using the Go command, however in your case you are building with bazel, so we either need a bazel fix or we need some other fix in the Go toolchain (possibly the cgo command).

@leitzler
Copy link
Contributor Author

leitzler commented Mar 6, 2023

Hmm, based on what I am seeing I think the "ldflags=-linkmode=external" is maybe not making it through to the Go command... the reason I say this is that the "loadelf" routines in question (which are generating the errors) are only invoked during internal linking.

Interesting, as a simple test I did change to -linkmode=foo and that makes it error out with the link help message, so it gets through in some way. Thanks for confirming that loadelf isn't invoked during internal linking, I'll see if I can find out why the external linkmode doesn't bite in my case.

I am working on a CL 471055 that will help solve these problems using the Go command, however in your case you are building with bazel, so we either need a bazel fix or we need some other fix in the Go toolchain (possibly the cgo command).

Yeah, as far as I've seen rules_go uses go test -cover and doesn't care about llvm coverage so I guess it could strip llvm coverage flags before compiling? It doesn't really feel like the cleanest solution though 😅

@gopherbot
Copy link

Change https://go.dev/cl/475375 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

@thanm thanm self-assigned this Mar 14, 2023
gopherbot pushed a commit that referenced this issue Mar 14, 2023
This patch changes the Go command to examine the set of compiler
flags feeding into the C compiler when packages that use cgo are built.
If any of a specific set of strange/dangerous flags are in use,
then the Go command generates a token file ("preferlinkext") and
embeds it into the compiled package's archive.

When the Go linker reads the archives of the packages feeding into the
link and detects a "preferlinkext" token, it will then use external
linking for the program by default (although this default can be
overridden with an explicit "-linkmode" flag).

The intent here is to avoid having to teach the Go linker's host object
reader to grok/understand the various odd symbols/sections/types that
can result from boutique flag use, but rather to just boot the objects
in question over to the C linker instead.

Updates #58619.
Updates #58620.
Updates #58848.

Change-Id: I56382dd305de8dac3841a7a7e664277826061eaa
Reviewed-on: https://go-review.googlesource.com/c/go/+/475375
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@thanm
Copy link
Contributor

thanm commented Mar 14, 2023

Update: I have submitted CL 475375, which should take care of the problem for "go test". If you can test this CL (or latest tip of go repo) to see if it fixes the problem, that would be great.

@leitzler
Copy link
Contributor Author

Super! I tested with go version devel go1.21-035db07d7c Tue Mar 14 13:17:46 2023 +0000 linux/amd64 and it do indeed fix this issue, thanks!

Are there any slim chances that CL 475375 gets back ported to 1.20? As it seems to prevent others from updating to Go 1.20 too.

@thanm
Copy link
Contributor

thanm commented Mar 14, 2023

Thanks for verifying.

I'll request a backport-- it will need to be reviewed/approved by the release team, however, so no guarantees.

@gopherbot
Copy link

Change https://go.dev/cl/476576 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

@gopherbot
Copy link

Change https://go.dev/cl/476577 mentions this issue: cmd/go,cmd/link: prefer external linking when strange cgo flags seen

@mknyszek mknyszek added this to the Backlog milestone Mar 15, 2023
gopherbot pushed a commit that referenced this issue Mar 17, 2023
… strange cgo flags seen

This patch changes the Go command to examine the set of compiler
flags feeding into the C compiler when packages that use cgo are built.
If any of a specific set of strange/dangerous flags are in use,
then the Go command generates a token file ("preferlinkext") and
embeds it into the compiled package's archive.

When the Go linker reads the archives of the packages feeding into the
link and detects a "preferlinkext" token, it will then use external
linking for the program by default (although this default can be
overridden with an explicit "-linkmode" flag).

The intent here is to avoid having to teach the Go linker's host object
reader to grok/understand the various odd symbols/sections/types that
can result from boutique flag use, but rather to just boot the objects
in question over to the C linker instead.

Fixes #59051.
Updates #58619.
Updates #58620.
Updates #58848.

Change-Id: I56382dd305de8dac3841a7a7e664277826061eaa
Reviewed-on: https://go-review.googlesource.com/c/go/+/475375
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit 035db07)
Reviewed-on: https://go-review.googlesource.com/c/go/+/476577
gopherbot pushed a commit that referenced this issue Mar 17, 2023
… strange cgo flags seen

This patch changes the Go command to examine the set of compiler
flags feeding into the C compiler when packages that use cgo are built.
If any of a specific set of strange/dangerous flags are in use,
then the Go command generates a token file ("preferlinkext") and
embeds it into the compiled package's archive.

When the Go linker reads the archives of the packages feeding into the
link and detects a "preferlinkext" token, it will then use external
linking for the program by default (although this default can be
overridden with an explicit "-linkmode" flag).

The intent here is to avoid having to teach the Go linker's host object
reader to grok/understand the various odd symbols/sections/types that
can result from boutique flag use, but rather to just boot the objects
in question over to the C linker instead.

Fixes #59050.
Updates #58619.
Updates #58620.
Updates #58848.

Change-Id: I56382dd305de8dac3841a7a7e664277826061eaa
Reviewed-on: https://go-review.googlesource.com/c/go/+/475375
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit 035db07)
Reviewed-on: https://go-review.googlesource.com/c/go/+/476576
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: All-But-Submitted
Development

No branches or pull requests

4 participants