test: TestReleaseUpgrade fails

[xiang90]

--- FAIL: TestReleaseUpgrade (1.24s)
    etcd_release_upgrade_test.go:78: error restarting etcd process (read /dev/ptmx: input/output error)
    etcd_release_upgrade_test.go:47: error closing etcd processes (exit status 1)

[gyuho]

@heyitsanthony I can only reproduce with concurrent run on this test.

go test -v -run TestReleaseUpgrade &
go test -v -run TestReleaseUpgrade

Have you ever seen Semaphore running tests concurrently?

Our test script runs only sequentially as far as I can tell.

https://github.com/coreos/etcd/blob/master/test#L72-L77

[heyitsanthony]

@gyuho e2e tests shouldn't run concurrently. Maybe something (TIME_WAIT connections?) is keeping the port from rebinding on etcdserver restart? Only the release upgrade test is doing one-by-one restart...

[gyuho]

This is most likely from the OS (reproducible with conflicting ports)

2016-08-08 12:28:21.139648 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8
2016-08-08 12:28:21.139786 C | etcdmain: listen tcp 127.0.0.1:20001: bind: address already in use
2016-08-08 12:28:21.139875 C | etcdmain: listen tcp 127.0.0.1:20003: bind: address already in use
--- FAIL: TestReleaseUpgrade (0.01s)
    etcd_release_upgrade_test.go:43: could not start etcd process cluster (read /dev/ptmx: input/output error)

The port might not have been freed up by the time we restart the cluster with the same port.

[xiang90]

@gyuho Do you print out logging by default? Anything we can do to improve it by avoiding the port issue?

[gyuho]

I will try what Anthony suggested

rebinding on etcdserver restart