net/http: MaxIdleConnsPerHost - Persistent connection limit

[gopherbot]

by surajn.vnit:

What does 'go version' print? go1.3 linux/amd64

What steps reproduce the problem?
If possible, include a link to a program on play.golang.org.
Setup -
- Front end HTTP server implemented using Go. It requests 1 backend service over HTTP.
Front end service waits for 10ms or the response from the backend service, whichever
earlier and sends response to the user.
Code - http://play.golang.org/p/rE0Q7N2nmT
- Back End Service - HTTP service implemented using Go. Accepts request, sleeps for 5ms
and responds back.
- Concurrency test: weighttp -n 10000 -c 2 -t 1  -k "http://127.0.0.1:8084/";
 
What happened?
1. Number of connections with the backend service in TIME_WAIT kept on increasing
(>5000 at the end of test)
2. Changed the MaxIdleConnsPerHost to 4. Number of connections to backend service is
constant at 4 and are in ESTABLISHED state.

What should have happened instead?
1. Ideally 2 to 4 persistent connections are sufficient to handle this much load. But
every connection after the limit MaxIdleConnsPerHost is reached, I think is not
persistent. This leads in too many connections being established and broken. This can in
turn bring the system down - which I am seeing at a little high concurrency.

This can be an issue for a service which have to communicate 100+ backend HTTP services,
as we cannot keep very high limit for MaxIdleConnsPerHost.

Case 2 is the expected behavior. So if MaxIdleConnsPerHost is sufficient, then there is
no issue.

Please provide any additional information below.

This can be the possible duplicate of
https://golang.org/issue/6785. Please check.
Golang-nuts link - https://groups.google.com/forum/#!topic/golang-nuts/-SLvF5OzOSI.

[bradfitz]

Comment 2:

I think this is probably a duplicate of issue #6785. But there's an easy workaround for
that: provide your own http.Transport.Dial func that has the policy you'd like. You'll
have to do a little bit of bookkeeping, but you can do exactly what you need.

Status changed to Duplicate.

Merged into issue #6785.

[bradfitz]

Comment 2:

I think this is probably a duplicate of issue #6785. But there's an easy workaround for
that: provide your own http.Transport.Dial func that has the policy you'd like. You'll
have to do a little bit of bookkeeping, but you can do exactly what you need.

Status changed to Duplicate.

Merged into issue #6785.