Sockets remain in TIME_WAIT / CLOSE_WAIT

[wereHamster]

I have a simple etcd setup with two machines in a cluster. Both servers eventually run out of free file descriptors (2013/09/10 16:47:32 http: Accept error: accept tcp [::]:7001: too many open files; retrying in 1s).

Etcd fails to properly close the sockets. They remain in TIME_WAIT / CLOSE_WAIT. You can use these two commands to inspect in which state the sockets are:

• lsof -a -p pidof etcd
• netstat -ntp

So far one of the etcd servers has leaked 14 file descriptors in about two hours.

[xiang90]

@wereHamster Based on your reporting on IRC, seems like you are not using the newest version of etcd, which has a socket leak problem (I am solving that).
I do not think etcd controls any tcp connection in the previous version. The golang's http package help to maintain all persistent connections. It is also possible that the leak is caused by the client side (based on the stats you gave me, there were about 10K reads). Which client library are you using?

[wereHamster]

The client is written in haskell and uses conduit-http for the http connection. I don't think the client is a problem though, as the dangling connections were on port 7001 between the two etcd instances.

[philips]

@wereHamster Can you please test with the latest master version? I don't know if this is fixed or how to reproduce.

[wereHamster]

I'm running v0.1.2-29-g3be69f0 now and the problem still persists. The server has about 2k connections in TIME_WAIT. Though etcd still accepts connections on port 4001, something which previously failed due to the 'out of sockets' condition. Also, I don't see any error messages in the log file.

So I'm not sure if it's fixed. One one hand etcd works (as in, clients can connect to it and do their job), on the other hand 2k connections in TIME_WAIT still seems suspicious.

[wereHamster]

Hm, only one of the two servers in the cluster was updated. The other one is still running v0.1.1-46-gadbcbef. That could explain that I'm seeing TIME_WAIT but no CLOSE_WAIT on the machine that's running the latest version. I'll update the second machine as well and will get back to you in a day or two.