net/http: TIME_WAIT problem for web server - go.http.Server

[gopherbot]

by dragos.constantin.stoica:

Hello,

First thank you for the great work that you are doing with go. We started to use go last
year for a medium sized project. Go is used as middleware for a 3 tier application and
acts as a web server. From this point of view it does a great job. We have a load of
about 300 users per week and about 20-30 concurrent users connected on the server
anytime of day from 08:00-17:00. The application is a document management - flow
management and the documents are stored in JSON format in CouchDB.

The problem:
When the load of 30 concurrent users per day is superseded the server enters in a state
where no more connections are accepted from users and no responses are sent to already
connected users. The go program does not crash.
The issues is linked with too many connections being in TIME_WAIT state, despite the
fact that the open file limit is not even reached.
If we kill the go program and restart it the problem is solved - TIME_WAIT connections
are freed and everything goes back to normal operation.

Which operating system are you using?
Ubuntu Server 11.04 64bits

Which version are you using?  (run 'go version')
1.0.3

Please provide any additional information below.
cat /proc/sys/fs/file-max 
70000

more /etc/sysctl.conf 
#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables
# See sysctl.conf (5) for information.
#

fs.file-max = 70000
...

more /etc/security/limits.conf 
# /etc/security/limits.conf
[user_for_go_program]       soft    nofile  10000
[user_for_go_program]       hard    nofile  30000
# End of file

[alberts]

Comment 1:

Are you closing all your http response bodies?

[remyoudompheng]

Comment 2:

Closing a connection makes it go in TIME_WAIT state. What part of your system creates
many TCP connections? I highly doubt it is the HTTP server, the problem is likely in
your internal pipes.

[gopherbot]

Comment 3 by dragos.constantin.stoica:

Hello,
I will doublecheck all responses from the server.
We are closing all connections to database server with:
defer resp.Body.Close()
where resp is a connection to CouchDB:
resp, err_get := conexiune.(*CouchPool.CouchConnection).HttpClient.Get(sURL)
but not all requests from the client are closed in this way.
Another source of this issue may be the ConnectionPool that we are using to access the
database. So we are currently investigating an alternative implementation of connection
sharing and reusing mechanism.
We also suspect that for short load bursts the system does not reclaim the sockets that
are in TIME_WAIT so this may lead the middleware to "hang".
Thank you for your answer.
I will leave this issue open since there is no diagnostic yet and no solution.

[remyoudompheng]

Comment 4:

The fact that sockets in TIME_WAIT are not reclaimed is normal. The TCP specification
requires that sockets go in TIME_WAIT state after being closed, usually for about a
minute. You cannot reuse a local port number until the socket is gone.
The effect of that is that you cannot open many connections from the same machine to the
same server in a short timeframe, it's not a Go problem, and it's not a file descriptor
problem either.

[ianlancetaylor]

Comment 5:

Marking this as WontFix since it's not clear that there is anything that can change on
the Go side.  TIME_WAIT is part of TCP and the kernel, not Go.
If you think that this is still a Go problem, please reopen and provide a test case. 
From the verbal description there is no Go problem, so we'll need to see some code.

Status changed to WontFix.