Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link, cmd/cgo: internal linking fails if CGO_CFLAGS enables LTO #43505

Open
mwhudson opened this issue Jan 5, 2021 · 9 comments
Open

cmd/link, cmd/cgo: internal linking fails if CGO_CFLAGS enables LTO #43505

mwhudson opened this issue Jan 5, 2021 · 9 comments

Comments

@mwhudson
Copy link
Contributor

@mwhudson mwhudson commented Jan 5, 2021

What version of Go are you using (go version)?

$ go version
go version devel +9eef49cfa6 Mon Jan 4 17:59:30 2021 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes, although the error is a bit different.

What operating system and processor architecture are you using (go env)?

Ubuntu 20.04 or 20.10 (current devel series).

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mwhudson/.cache/go-build"
GOENV="/home/mwhudson/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mwhudson/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mwhudson/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mwhudson/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mwhudson/src/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel +9eef49cfa6 Mon Jan 4 17:59:30 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1258774523=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran a trivial os/user-using program with CFLAGS that caused gcc to create object files ready for link time optimization:

mwhudson@anduril:~/tmp$ cat user.go
package main

import (
	"fmt"
	"os/user"
)

func main() {
	u, _ := user.Current()
	fmt.Printf("username: %s\n", u.Name)
}
mwhudson@anduril:~/tmp$ CGO_CFLAGS='-g -flto -ffat-lto-objects' ~/src/go/bin/go run user.go 

What did you expect to see?

username: Michael Hudson-Doyle

What did you see instead?

Errors of the form "loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x001.o): 67693: sym#26: ignoring symbol in section 5 (type 0)" and then a panic:

# command-line-arguments
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x001.o): 67693: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x002.o): 67704: sym#44: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x003.o): 67712: sym#32: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/1c/1cfde03a90c420f612c840a344f1de135898b62010e420c7b14c3e353c13c3d9-d(_x004.o): 67719: sym#29: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x001.o): 67723: sym#24: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x002.o): 67730: sym#29: ignoring symbol in section 4 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x003.o): 67735: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x004.o): 67741: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x005.o): 67751: sym#38: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x006.o): 67759: sym#31: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x007.o): 67764: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x008.o): 67769: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x009.o): 67774: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x010.o): 67779: sym#26: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x011.o): 67785: sym#27: ignoring symbol in section 5 (type 0)
loadelf: /home/mwhudson/.cache/go-build/f8/f82d697d7b7630e244251862c642563a76958f85413b4848809324ae1c92be56-d(_x012.o): 67790: sym#34: ignoring symbol in section 5 (type 0)
_cgo_callers: relocation target x_cgo_callers not defined
_cgo_init: relocation target x_cgo_init not defined
_cgo_mmap: relocation target x_cgo_mmap not defined
_cgo_munmap: relocation target x_cgo_munmap not defined
_cgo_notify_runtime_init_done: relocation target x_cgo_notify_runtime_init_done not defined
/home/mwhudson/src/go/pkg/tool/linux_amd64/link: too many errors
unexpected fault address 0x7f310460c000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f310460c000 pc=0x46db13]

goroutine 129 [running]:
runtime.throw(0x6dac2e, 0x5)
/home/mwhudson/src/go/src/runtime/panic.go:1112 +0x72 fp=0xc00061fd18 sp=0xc00061fce8 pc=0x436e12
runtime.sigpanic()
/home/mwhudson/src/go/src/runtime/signal_unix.go:737 +0x268 fp=0xc00061fd50 sp=0xc00061fd18 pc=0x44e4a8
runtime.memmove(0x7f310460baa0, 0x7f3104eb29b1, 0xa6a)
/home/mwhudson/src/go/src/runtime/memmove_amd64.s:367 +0x433 fp=0xc00061fd58 sp=0xc00061fd50 pc=0x46db13
cmd/link/internal/ld.(*OutBuf).WriteSym(0xc00111e000, 0xc000128000, 0x3925, 0x2, 0xc0000b60c5, 0x1)
/home/mwhudson/src/go/src/cmd/link/internal/ld/outbuf.go:300 +0x2bc fp=0xc00061fde8 sp=0xc00061fd58 pc=0x5ea7fc
cmd/link/internal/ld.writeBlock(0xc00010e000, 0xc00111e000, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x40daa0, 0x9b4a5, 0xc0000b60c5, 0x1, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:952 +0x1a9 fp=0xc00061ff10 sp=0xc00061fde8 pc=0x589149
cmd/link/internal/ld.writeBlocks.func1(0xc00010e000, 0xc001114020, 0xc0000f40e0, 0xc00111e000, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x401000, 0x9b4a5, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:909 +0xb1 fp=0xc00061ff78 sp=0xc00061ff10 pc=0x608311
runtime.goexit()
/home/mwhudson/src/go/src/runtime/asm_amd64.s:1367 +0x1 fp=0xc00061ff80 sp=0xc00061ff78 pc=0x46c901
created by cmd/link/internal/ld.writeBlocks
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:908 +0x565

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0011a4038)
/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0011a4030)
/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65
cmd/link/internal/ld.asmb(0xc00010e000)
/home/mwhudson/src/go/src/cmd/link/internal/ld/asmb.go:73 +0x235
cmd/link/internal/ld.Main(0x87d2e0, 0x20, 0x20, 0x1, 0x7, 0x10, 0x0, 0x0, 0x6e663e, 0x1b, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/main.go:331 +0x140e
main.main()
/home/mwhudson/src/go/src/cmd/link/main.go:68 +0x258

goroutine 24 [semacquire]:
sync.runtime_Semacquire(0xc001230008)
/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc001230000)
/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65
cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119eb60, 0xc0000f40e0, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x53f000, 0x15ca0, 0x8bd260, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:924 +0x736
cmd/link/internal/ld.writeDatblkToOutBuf(...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:1006
cmd/link/internal/ld.datblk(0xc00010e000, 0xc00119eb60, 0x53f000, 0x15ca0)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:994 +0xad
cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a8, 0xc00010e000, 0xc00119eb60, 0x53f000, 0x15ca0)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a
created by cmd/link/internal/ld.writeParallel
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152

goroutine 23 [runnable]:
cmd/link/internal/loader.(*Loader).SubSym(...)
/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:1740
cmd/link/internal/loader.(*Loader).AttrSubSymbol(...)
/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:1161
cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119eaf0, 0xc0000f40e0, 0xc000128000, 0xc0011b6008, 0x1d27, 0x1d27, 0x49d000, 0xa1028, 0x8bd260, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:858 +0x785
cmd/link/internal/ld.writeDatblkToOutBuf(...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:1006
cmd/link/internal/ld.datblk(0xc00010e000, 0xc00119eaf0, 0x49d000, 0xa1028)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:994 +0xad
cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a8, 0xc00010e000, 0xc00119eaf0, 0x49d000, 0xa1028)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a
created by cmd/link/internal/ld.writeParallel
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152

goroutine 21 [semacquire]:
sync.runtime_Semacquire(0xc001114028)
/home/mwhudson/src/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc001114020)
/home/mwhudson/src/go/src/sync/waitgroup.go:130 +0x65
cmd/link/internal/ld.writeBlocks(0xc00010e000, 0xc00119ea10, 0xc0000f40e0, 0xc000128000, 0xc000eae000, 0x5e2, 0x6a0, 0x401000, 0x9b4a5, 0xc0000b60c5, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:924 +0x736
cmd/link/internal/ld.CodeblkPad(...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:834
cmd/link/internal/ld.asmb.func1(0xc00010e000, 0xc00119ea10, 0x401000, 0x9b4a5)
/home/mwhudson/src/go/src/cmd/link/internal/ld/asmb.go:39 +0xd3
cmd/link/internal/ld.writeParallel.func1(0xc0011a4030, 0x6f23a0, 0xc00010e000, 0xc00119ea10, 0x401000, 0x9b4a5)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:988 +0x7a
created by cmd/link/internal/ld.writeParallel
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:986 +0x152

goroutine 113 [runnable]:
os.Exit(0x2)
/home/mwhudson/src/go/src/os/proc.go:62 +0x73
cmd/link/internal/ld.Exit(0x2)
/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:31 +0x8a
cmd/link/internal/ld.Exitf(0x6dfa23, 0xf, 0x0, 0x0, 0x0)
/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:38 +0x106
cmd/link/internal/ld.afterErrorAction()
/home/mwhudson/src/go/src/cmd/link/internal/ld/util.go:49 +0x7a
cmd/link/internal/loader.(*ErrorReporter).Errorf(0xc00010e010, 0xa5e0, 0x6e8fde, 0x20, 0xc001225ae0, 0x1, 0x1)
/home/mwhudson/src/go/src/cmd/link/internal/loader/loader.go:2619 +0x143
cmd/link/internal/ld.(*ErrorReporter).errorUnresolved(0xc00010e010, 0xc000128000, 0xa5e0, 0xa5df)
/home/mwhudson/src/go/src/cmd/link/internal/ld/errors.go:63 +0x4e9
cmd/link/internal/ld.(*relocSymState).relocsym(0xc001225ea0, 0xa5e0, 0x7f310474b4c0, 0x8, 0x7b40)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:199 +0x338a
cmd/link/internal/ld.writeBlock(0xc00010e000, 0xc001232000, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x54d4c0, 0x15ca0, 0x8bd260, 0x200, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:953 +0x1e7
cmd/link/internal/ld.writeBlocks.func1(0xc00010e000, 0xc001230000, 0xc0000f40e0, 0xc001232000, 0xc000128000, 0xc0011c2218, 0x4e5, 0x4e5, 0x53f000, 0x15ca0, ...)
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:909 +0xb1
created by cmd/link/internal/ld.writeBlocks
/home/mwhudson/src/go/src/cmd/link/internal/ld/data.go:908 +0x565

@mwhudson
Copy link
Contributor Author

@mwhudson mwhudson commented Jan 5, 2021

So what's going on here is that these cflags make gcc include symbols in object files that cmd/link/internal/loadelf doesn't like, causing it to abandon parsing the file and so not find symbols it needs to find.

loadelf could be taught to ignore these symbols, I guess, or just skip all symbols it doesn't know how to handle, but really with my Ubuntu/Debian golang maintainer hat on, I'd rather there was some way of preventing the linker ever attempt internal linking if any cgo is used at all, at least during package builds. I don't want to disable internal linking entirely -- that would mean all go executables, even those not using cgo at all, would be dynamically linked, which seems a bit much of a change. I guess I can do this by patching internalpkg in cmd/link/internal/ld/lib.go to be empty but that also seems a bit much. I don't think there is an existing flag or environment variable that does this.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 5, 2021

We want to use internal linking for Go programs that do not explicitly use cgo. That includes Go programs that use the standard packages net and os/user, even though those packages do use cgo by default. So we do want internal linking to work for programs that use cgo, provided the only packages that use cgo are those in the standard library.

But we could consider using external linking by default if CGO_CFLAGS contains anything other than -g and -O options, or something along those lines.

@mwhudson
Copy link
Contributor Author

@mwhudson mwhudson commented Jan 5, 2021

That would work. Can the linker assume that CGO_CFLAGS in its environment is the one that was used to build the object files it is processing though? The cflags are recorded in the _cgo_flags but that's only used by the gccgo support in cmd/go currently, I'm not sure the linker can read them from anywhere?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 5, 2021

It should be the case that if the linker sees CGO_CFLAGS in the environment, then it was used to build the object files.

But what about flags specified with #cgo CFLAGS:? It seems to me that they ought to have an identical effect. And I don't know if the linker can see those at present.

@mwhudson
Copy link
Contributor Author

@mwhudson mwhudson commented Jan 5, 2021

It should be the case that if the linker sees CGO_CFLAGS in the environment, then it was used to build the object files.

It's certainly good enough for my use case!

But what about flags specified with #cgo CFLAGS:? It seems to me that they ought to have an identical effect. And I don't know if the linker can see those at present.

I wondered about that, but any use of cgo outside the standard library triggers external linking anyway so this isn't an issue (I think we can be sure no-one will accidentally add "#cgo CFLAGS: -flto -ffat-lto-objects" to runtime/cgo ...)

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 5, 2021

Good point.

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 5, 2021

Change https://golang.org/cl/281314 mentions this issue: cmd/link: check CGO_CFLAGS for non -g/-O options before internal linking

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Jan 11, 2021

I think only accepting -g and -O is too narrow. On some platforms there are some flags we use. For example, -march=armv6 is used when we build ARMv6 release. On Darwin we pass some flag to specify the macOS version. Listing all allowed flags seems complicated. Compared to this, maybe -linkmode=external-if-cgo is more desirable.

@gopherbot
Copy link

@gopherbot gopherbot commented May 25, 2021

Change https://golang.org/cl/322614 mentions this issue: cmd/link, cmd/cgo: support -flto in CFLAGS

gopherbot pushed a commit that referenced this issue Jun 1, 2021
The linker now accepts unrecognized object files in external linking mode.
These objects will simply be passed to the external linker.
This permits using -flto which can generate pure byte code objects,
whose symbol table the linker does not know how to read.

The cgo tool now passes -fno-lto when generating objects whose symbols
it needs to read. The cgo tool now emits matching types in different
objects, so that the lto linker does not report a mismatch.

This is based on https://golang.org/cl/293290 by Derek Parker.

For #43505
Fixes #43830
Fixes #46295

Change-Id: I6787de213417466784ddef5af8899e453b4ae1ad
Reviewed-on: https://go-review.googlesource.com/c/go/+/322614
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Hudson-Doyle <michael.hudson@canonical.com>
@seankhliao seankhliao changed the title if CGO_CFLAGS enables LTO, internal linking fails cmd/link, cmd/cgo: internal linking fails if CGO_CFLAGS enables LTO Jun 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants