Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misc/cgo/testshared: segfaults on linux/arm64 Fedora 29 #31823

Closed
jcajka opened this issue May 3, 2019 · 12 comments

Comments

Projects
None yet
4 participants
@jcajka
Copy link
Contributor

commented May 3, 2019

What version of Go are you using (go version)?

go version devel +004fb5cb8d Fri May 3 03:49:11 2019 +0000 linux/arm64

Does this issue reproduce with the latest release?

Does reproduce on master, Go 1.12 and Go 1.11 on Fedora 29.

What operating system and processor architecture are you using (go env)?

linux/arm64

What did you do?

I have built master branch with go1.11.6 on linux/arm64 host(Fedora 29) and

go tool dist test -v -v -v -k

This is with

gcc-8.3.1-2.fc29.aarch64
binutils-2.31.1-17.fc29.aarch64
glibc-2.28-27.fc29.aarch64

What did you expect to see?

Tests passing

What did you see instead?

##### ../misc/cgo/testshared
# go tool dist test -run=^testshared$
go test -short -count=1 -tags= -run= -timeout=10m0s
--- FAIL: TestTrivialExecutable (8.15s)
    shared_test.go:41: executing ../../bin/trivial (trivial executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestDivisionExecutable (3.22s)
    shared_test.go:41: executing ../../bin/division (division executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestCgoExecutable (6.45s)
    shared_test.go:41: executing ../../bin/execgo (cgo executable) failed signal: segmentation fault (core dumped):
--- FAIL: TestGopathShlib (10.57s)
    shared_test.go:41: executing ../../bin/exe (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestTwoGopathShlibs (11.90s)
    shared_test.go:41: executing ../../bin/exe2 (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestThreeGopathShlibs (13.34s)
    shared_test.go:41: executing ../../bin/exe3 (executable linked to GOPATH library) failed signal: segmentation fault (core dumped):
--- FAIL: TestABIChecking (6.29s)
    shared_test.go:885: exe failed, but without line "abi mismatch detected between the executable and lib"; got output:
--- FAIL: TestImplicitInclusion (3.27s)
    shared_test.go:41: executing ../../bin/implicitcmd (running executable linked against library that contains same package as it) failed signal: segmentation fault (core dumped):
--- FAIL: TestInterface (4.50s)
    shared_test.go:41: executing ../../bin/iface (running type/itab uniqueness tester) failed signal: segmentation fault (core dumped):
--- FAIL: TestGlobal (3.58s)
    shared_test.go:41: executing ../../bin/global (global executable) failed signal: segmentation fault (core dumped):
shared_test.go:72: executing go test -installsuffix=6129484611666145821 -linkshared -test.short sync/atomic failed exit status 1:
exit status 1
FAIL	misc/cgo/testshared	111.018s
2019/05/03 10:43:35 Failed: exit status 1
run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std
run: go list -gcflags=all= -ldflags=all= -f={{if .Stale}}	STALE {{.ImportPath}}: {{.StaleReason}}{{end}} std DONE

@ianlancetaylor ianlancetaylor changed the title cgo testshared segfults on linux/arm64 Fedora 29 misc/cgo/testshared: segfults on linux/arm64 Fedora 29 May 3, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

Odd. The builder isn't reporting any problems.

Can you reproduce the problem cd misc/cgo/testshared; go test -v? What is the output?

@ianlancetaylor ianlancetaylor added this to the Go1.13 milestone May 3, 2019

@jcajka jcajka changed the title misc/cgo/testshared: segfults on linux/arm64 Fedora 29 misc/cgo/testshared: segfaults on linux/arm64 Fedora 29 May 6, 2019

@jcajka

This comment has been minimized.

Copy link
Contributor Author

commented May 7, 2019

Yeah, it looks odd. Possibly something wrong with the binaries/runtime. Build/test log looks sane to me, although failing. This unfortunately flew under my radar for some time :(.
Output of cd misc/cgo/testshared; go test -c -v; ./testshared.test -test.v
About to dig in to the binaries built during the test.

For the record what are the versions of binutils, glibc and gcc on builders? To rule out that angle.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

CC @bradfitz: see question about tools versions on builders. Thanks.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

Actually, never mind, I can use gomote to find the tool versions.

We're using gcc 4.9.2, binutils 2.25, glibc 2.19-18. So quite a bit older than the tools you are using. We are running Debian jessie.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

Thanks for sending the log output. Unfortunately I agree that it all looks normal. My first guess would of course be that the newer tools are causing a problem somehow, but I don't know why that would be. I think we'll need some way to reproduce the problem ourselves.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2019

I can add a Fedora builder, but in the meantime (still commuting) I just updated our debian-sid builder. It hadn't been updated in quite some time, and it gcc was pretty old.

From 7.3.0 to 8.3.0:

bradfitz@go:~/src/golang.org/x/build/env/linux-x86-sid$ docker run -ti gcr.io/symbolic-datum-552/linux-x86-sid:latest /bin/bash
root@759b9aca4e34:/# gcc --version
gcc (Debian 7.3.0-18) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

bradfitz@go:~/src/golang.org/x/build/env/linux-x86-sid$ docker run -ti golang/linux-x86-sid /bin/bash
root@a6236231b605:/# gcc --version
gcc (Debian 8.3.0-7) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Just deployed now.

@gopherbot

This comment has been minimized.

Copy link

commented May 7, 2019

Change https://golang.org/cl/175737 mentions this issue: env/linux-x86-sid: update our sid builder

@jcajka

This comment has been minimized.

Copy link
Contributor Author

commented May 7, 2019

@ianlancetaylor Just did quick poke on it with gdb and the crash seems to happen during executable startup prior main function is invoked.

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000ffffbf2c8d24 in __libc_start_main () from /lib64/libc.so.6
#2  0x0000000000400a34 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Intriguing is that the shared pie tests pass just fine.

To note I will continue digging in to this next week as I will be mostly offline for next 7 days.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

@bradfitz Note that this bug is for arm64. I can't tell whether you updated the arm64 builder or not.

gopherbot pushed a commit to golang/build that referenced this issue May 7, 2019

env/linux-x86-sid: update our sid builder
Updates golang/go#31823

Change-Id: I96d83cf7a9994c8ed6d0ea58af038577d48ae737
Reviewed-on: https://go-review.googlesource.com/c/build/+/175737
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2019

Note that this bug is for arm64. I can't tell whether you updated the arm64 builder or not.

I just did linux-amd64-sid out of hope that updated tools would let us reproduce it alone. And we try to update the sid builder regularly anyway, so doesn't hurt.

But sadly that didn't catch it. We do have enough arm64 capacity that we could run a Fedora arm64 Docker container on it, assuming that'd be enough and it's not a kernel issue. That's a bit more tedious so I likely won't get to it right away.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 8, 2019

I also tried Fedora 30 (also amd64) on a whim but it passed fine. I didn't try Fedora 29 on amd64, though.

@jcajka

This comment has been minimized.

Copy link
Contributor Author

commented May 21, 2019

More digging in, this is actually dupe of #28334 and it has been/is binutils/gold issue https://sourceware.org/bugzilla/show_bug.cgi?id=23870 fixed on the trunk.
So closing. Thanks to nickc for quick backport of the patch for all Fedora releases.

@jcajka jcajka closed this May 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.