Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

net/http: BenchmarkClientServerParallel4 hangs and is useless #20166

Open
bradfitz opened this Issue Apr 28, 2017 · 4 comments

Comments

Projects
None yet
4 participants
Owner

bradfitz commented Apr 28, 2017

It seems the BenchmarkClientServerParallel4 benchmark sometimes hangs while running. Sometimes it needs to be killed (first example) and sometimes it recovers on its own, leading to useless numbers.

In any case, I can't get good numbers out of it.

I suspect it's consuming too many sockets and running out of addresses or something.

bradfitz@gdev:~/go/src/net/http$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: linux
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4           30000             39470 ns/op            7539 B/op         73 allocs/op
BenchmarkClientServerParallel4-4           50000             40934 ns/op            7563 B/op         73 allocs/op
BenchmarkClientServerParallel4-4        ^C
bradfitz@gdev:~/go/src/net/http$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: linux
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4            5000            283132 ns/op            7244 B/op         85 allocs/op
BenchmarkClientServerParallel4-4               1        7011289330 ns/op           54720 B/op        527 allocs/op
BenchmarkClientServerParallel4-4               1        7010777787 ns/op           37472 B/op        238 allocs/op
PASS
ok      net/http        16.585s

@bradfitz bradfitz added this to the Unplanned milestone Apr 28, 2017

Owner

bradfitz commented Apr 28, 2017

(At d286399, on linux/amd64)

Contributor

kennygrant commented Jul 14, 2017

Might be related to #18753 with BenchmarkClientServerParallel64 on mac os - which is the same test just with more parallelism. I've see a similar problems on mac os with BenchmarkClientServerParallel4 and variants. The first 3 runs are fine, second time it is run thrice it dies with the error: can't assign requested address. Waiting a minute and trying again results in the same pattern - fine for first run, fails afterward, as opposed to trying again straight away which fails repeatedly, so it does seem to be exhausting resources by tying them up for too long.

netstat -a after the test shows thousands of connections in state TIME_WAIT for a minute or two, but I guess that's normal?

kenny$ cd src/net/http
kenny$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: darwin
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4   	   30000	     51827 ns/op	    7677 B/op	      75 allocs/op
BenchmarkClientServerParallel4-4   	   10000	    156941 ns/op	    7948 B/op	      76 allocs/op
BenchmarkClientServerParallel4-4   	   10000	    149905 ns/op	    7936 B/op	      76 allocs/op
PASS
ok  	net/http	5.105s
kenny$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: darwin
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4   	   10000	    284811 ns/op	    7809 B/op	      76 allocs/op
--- BENCH: BenchmarkClientServerParallel4-4
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address

CL https://golang.org/cl/49031 mentions this issue.

xiam commented Jul 16, 2017

I think this problem is caused by how are we using ts.Client().

httptest.Server has a Client function that returns the same client every time it's called.

When we use ts.Client() inside a parallel test without calling CloseIdleConnections() on the transport, a lot of idle connections start to pile up and the OS may decide not to grant any one else more.

httpserver.Server does close the transport's idle connections, but only does so when the server itself is closed.

I submitted a solution that I think fits with the httptest.Server's design, but this could also be fixed by using something like:

tr := &Transport{}
defer tr.CloseIdleConnections()
c := &Client{Transport: tr}

instead of c = ts.Client()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment