New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: segfault during test using -coverpkg and -race. #28109

Open
gaillard opened this Issue Oct 10, 2018 · 7 comments

Comments

Projects
None yet
5 participants
@gaillard

gaillard commented Oct 10, 2018

What version of Go are you using (go version)?

go1.11.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Linux amd64

What did you do?

go test -c -o aTestBinary -coverpkg package1,package2,package3 -covermode atomic -race -ldflags '-w -s -extldflags "-static -Wl,--no-export-dynamic"'
./aTestBinary -test.v -test.run . -test.coverprofile coverout

What did you expect to see?

Success

What did you see instead?

unexpected fault address 0x19600
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x19600 pc=0x12a0389]

goroutine 63887 [running]:
runtime: unexpected return pc for runtime.throw called from 0x5
stack: frame={sp:0x7f0386ffca70, fp:0x7f0386ffcaa0} stack=[0xc0021f4000,0xc0021f6000)

runtime.throw(0x5, 0x7f0386ffcae0)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x7f0386ffcaa0 sp=0x7f0386ffca70 pc=0x497d42
created by ...

goroutine 1 [chan receive]:
testing.tRunner.func1(0xc000112100)
	/usr/local/go/src/testing/testing.go:803 +0x2e4
testing.tRunner(0xc000112100, 0xc00011bc98)
	/usr/local/go/src/testing/testing.go:831 +0x17f
testing.runTests(0xc0000c4060, 0x410d5c0, 0xa5, 0xa5, 0xc0000bc000)
	/usr/local/go/src/testing/testing.go:1117 +0x4ef
testing.(*M).Run(0xc000232200, 0x0)
	/usr/local/go/src/testing/testing.go:1034 +0x2ef
.../....TestMain(0xc000232200)
	/go/src/....go:36 +0x123
main.main()
	_testmain.go:962 +0x33f

goroutine 50 [chan receive]:
testing.(*testContext).waitParallel(0xc0000ad1d0)
	/usr/local/go/src/testing/testing.go:926 +0x15c
testing.(*T).Parallel(0xc000112200)
	/usr/local/go/src/testing/testing.go:733 +0x3e5
...

Probably related #23882?

@agnivade agnivade changed the title from Segfault during test using -coverpkg and -race. to cmd/go: segfault during test using -coverpkg and -race. Oct 10, 2018

@agnivade agnivade added this to the Go1.12 milestone Oct 10, 2018

@agnivade

This comment has been minimized.

Member

agnivade commented Oct 10, 2018

Do you have a shortened reproducer or perhaps a sample repo for us to test this out and reproduce it ourselves ?

@ianlancetaylor ianlancetaylor changed the title from cmd/go: segfault during test using -coverpkg and -race. to runtime: segfault during test using -coverpkg and -race. Oct 10, 2018

@gaillard

This comment has been minimized.

gaillard commented Oct 11, 2018

I don't have a repro as it happens in the middle of a larger suite with a number of parallel tests running. However it appears to be related mainly to the race detector. I have not seen the faults without the race detector and without the coverpkg option. For example here is another one, with race detector but without any coverage:

unexpected fault address 0x2600
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2600 pc=0x12629d9]

goroutine 424974 [running]:
runtime: unexpected return pc for runtime.throw called from 0x6
stack: frame={sp:0x7f0bf7ffea70, fp:0x7f0bf7ffeaa0} stack=[0xc002452000,0xc00245a000)

runtime.throw(0x6, 0x7f0bf7ffeae0)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x7f0bf7ffeaa0 sp=0x7f0bf7ffea70 pc=0x497d42
created by testing.(*B).RunParallel
	/usr/local/go/src/testing/benchmark.go:619 +0x256
@gaillard

This comment has been minimized.

gaillard commented Oct 15, 2018

@agnivade Anyway I can narrow down the PC address to help point in a right direction?

@agnivade

This comment has been minimized.

Member

agnivade commented Oct 16, 2018

/cc @ianlancetaylor for further comments.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 16, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Oct 16, 2018

The signal report is giving you a PC address. Can you tell us what is at that address?

@dballard

This comment has been minimized.

dballard commented Nov 16, 2018

go 1.10.3
I am incorporating https://github.com/cretz/bine into https://git.openprivacy.ca/cwtch.im/cwtch/

~/s/g/s/c/c/peer$ go test
PASS
ok  	cwtch.im/cwtch/peer	0.003s

~/s/g/s/c/c/peer$ go test -coverprofile=peer.cover.out
PASS
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x647958]

goroutine 23 [running]:
github.com/cretz/bine/tor.(*Tor).EnableNetwork(0x0, 0x73e520, 0xc4200a4050, 0x1, 0x0, 0x0)
	/home/dan/src/go/src/github.com/cretz/bine/tor/tor.go:367 +0x78
github.com/cretz/bine/tor.(*Tor).Dialer(0x0, 0x0, 0x0, 0xc420132e38, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/github.com/cretz/bine/tor/dialer.go:55 +0x3c3
git.openprivacy.ca/openprivacy/libricochet-go/utils.(*NetworkResolver).Resolve(0xc420041ed8, 0x0, 0xc4201025c0, 0x38, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/git.openprivacy.ca/openprivacy/libricochet-go/utils/networkresolver.go:44 +0x22c
git.openprivacy.ca/openprivacy/libricochet-go.Open(0x0, 0xc4201025c0, 0x38, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/git.openprivacy.ca/openprivacy/libricochet-go/ricochet.go:18 +0x56
cwtch.im/cwtch/peer/connections.(*PeerPeerConnection).Run(0xc4200a87d0, 0x38, 0x8c7760)
	/home/dan/src/go/src/cwtch.im/cwtch/peer/connections/peerpeerconnection.go:113 +0x5e
created by cwtch.im/cwtch/peer/connections.(*Manager).ManagePeerConnection
	/home/dan/src/go/src/cwtch.im/cwtch/peer/connections/connectionsmanager.go:39 +0x17b
exit status 2
FAIL	cwtch.im/cwtch/peer	0.009s

It seems to be a problem that my Init function which spawns a go thread takes a *tor.Tor argument, which for these tests is never used (just testing paths where we don't make connections, so it's never invoked) so I pass in a nil. The test runs fine, it seems the cover is attempting to follow the uncalled code

test

err = alice.TrustPeer(bobOnion)
if err == nil {
	t.Errorf("trust peer should fail since alice does not know about bob")
}
func (cp *cwtchPeer) TrustPeer(peer string) error {
	err := cp.Profile.TrustPeer(peer)
	if err == nil {
		cp.PeerWithOnion(peer)
	}
	return err
}

PeerWithOnion should never be called, and the test passes, but as you can see from the segfault trace, it has tracked into PeerWithOnion which calls ManagePeerConections which starts a goroutine go ppc.Run() which is where the above stack trace starts. Then it decends into our dependancy and segfaults.

What is really weird is since we've supplied nil it... shoudn't go that far? but the cover check decends into the tor.Tor object ok, but then it seems to be at

	if vals, err := t.Control.GetConf("DisableNetwork"); err != nil {

when the tor.Tor (our nil) object tries to access a field it finally explodes and fails

hope this is helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment