consensus: non-deterministic test failure

[ebuchman]

Perhaps we're not shutting down a switch correctly

--- FAIL: TestReactorValidatorSetChanges (0.15s)
panic: listen tcp 127.0.0.1:4182: bind: address already in use [recovered]
	panic: listen tcp 127.0.0.1:4182: bind: address already in use

goroutine 7477 [running]:
testing.tRunner.func1(0xc4214a9ef0)
	/usr/local/go/src/testing/testing.go:742 +0x567
panic(0xf51260, 0xc424c99ef0)
	/usr/local/go/src/runtime/panic.go:502 +0x24a
github.com/tendermint/tendermint/p2p.MakeSwitch(0xc4202248c0, 0x0, 0x102c997, 0x9, 0x102d928, 0xb, 0xc42189f878, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/tendermint/tendermint/p2p/test_util.go:182 +0xc71
github.com/tendermint/tendermint/p2p.MakeConnectedSwitches(0xc4202248c0, 0x7, 0xc42189f878, 0x1182e60, 0x0, 0x0, 0x0)
	/go/src/github.com/tendermint/tendermint/p2p/test_util.go:75 +0x10c
github.com/tendermint/tendermint/consensus.startConsensusNet(0xc4214a9ef0, 0xc4213b36c0, 0x7, 0x7, 0x7, 0x1182d18, 0xc4213b36c0, 0x7, 0x7, 0xc4228bdd00, ...)
	/go/src/github.com/tendermint/tendermint/consensus/reactor_test.go:56 +0x6aa
github.com/tendermint/tendermint/consensus.TestReactorValidatorSetChanges(0xc4214a9ef0)
	/go/src/github.com/tendermint/tendermint/consensus/reactor_test.go:339 +0x113
testing.tRunner(0xc4214a9ef0, 0x1182c10)
	/usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x565
FAIL	github.com/tendermint/tendermint/consensus	4.409s
Exited with code 1

[joe-bowman]

For reference, I have had the same issue here today:

--- FAIL: TestReactorProposalHeartbeats (0.00s)
panic: listen tcp 127.0.0.1:64813: bind: address already in use [recovered]
	panic: listen tcp 127.0.0.1:64813: bind: address already in use

goroutine 1765 [running]:
testing.tRunner.func1(0xc420c8f770)
	/usr/lib/go/src/testing/testing.go:742 +0x29f
panic(0x5577bc818fa0, 0xc421b9e690)
	/usr/lib/go/src/runtime/panic.go:502 +0x22d
github.com/tendermint/tendermint/p2p.MakeSwitch(0xc4201ca8c0, 0x1, 0x5577bc1ba996, 0x9, 0x5577bc1bb944, 0xb, 0xc421519d58, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/tendermint/tendermint/p2p/test_util.go:182 +0x80e
github.com/tendermint/tendermint/p2p.MakeConnectedSwitches(0xc4201ca8c0, 0x4, 0xc421519d58, 0x5577bc8ef470, 0x0, 0x0, 0x0)
	/go/src/github.com/tendermint/tendermint/p2p/test_util.go:75 +0xfb
github.com/tendermint/tendermint/consensus.startConsensusNet(0xc420c8f770, 0xc421b37e80, 0x4, 0x4, 0x4, 0xc421519f90, 0x1, 0x1, 0xc421b37e80, 0x4, ...)
	/go/src/github.com/tendermint/tendermint/consensus/reactor_test.go:56 +0x458
github.com/tendermint/tendermint/consensus.TestReactorProposalHeartbeats(0xc420c8f770)
	/go/src/github.com/tendermint/tendermint/consensus/reactor_test.go:223 +0xfc
testing.tRunner(0xc420c8f770, 0x5577bc8ef200)
	/usr/lib/go/src/testing/testing.go:777 +0xd2
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:824 +0x2e2

Seems like the same underlying cause.

[ebuchman]

Seen again:

panic: Failed to listen on 0.0.0.0:33074: listen tcp 0.0.0.0:33074: bind: address already in use

goroutine 1 [running]:
github.com/tendermint/tendermint/rpc/test.StartTendermint(0x1452320, 0xc0000309c0, 0x11814a0)
	/go/src/github.com/tendermint/tendermint/rpc/test/helpers.go:104 +0x103
github.com/tendermint/tendermint/rpc/client_test.TestMain(0xc000854380)
	/go/src/github.com/tendermint/tendermint/rpc/client/main_test.go:17 +0x4d
main.main()
	_testmain.go:138 +0x32f
FAIL	github.com/tendermint/tendermint/rpc/client	0.104s
Exited with code 1

[tac0turtle]

closing this, I haven't seen this error. If it comes up again, we can open a new issue and assign it accordingly.