unix sockets consumes too many file descriptors

[vixns]

Running puppetmaster with puma, it quickly run out of files descriptors when binding unix socket.

strace output sample :

[...]
[pid 30966] fcntl64(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 30966] accept(5, 0xb6375a5e, [110]) = -1 EMFILE (Too many open files)
[pid 30966] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 30966] select(8, [5 7], NULL, NULL, NULL) = 1 (in [5])
[pid 30966] fcntl64(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 30966] accept(5, 0xb6375a5e, [110]) = -1 EMFILE (Too many open files)
[pid 30966] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 30966] select(8, [5 7], NULL, NULL, NULL) = 1 (in [5])
[pid 30966] fcntl64(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
[pid 30966] accept(5, 0xb6375a5e, [110]) = -1 EMFILE (Too many open files)
[pid 30966] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[...]

lsof output sample :

[...]
puma 30941 6180 puppet 146u unix 0xe246f0c0 0t0 10027913 /var/run/puppet/master.sock
puma 30941 6180 puppet 147u unix 0xe87c0a40 0t0 10027791 /var/run/puppet/master.sock
puma 30941 6180 puppet 148u unix 0xe258ea40 0t0 10027251 /var/run/puppet/master.sock
puma 30941 6180 puppet 149u unix 0xe88c62c0 0t0 10027260 /var/run/puppet/master.sock
puma 30941 6180 puppet 150u unix 0xe246f5c0 0t0 10027914 /var/run/puppet/master.sock
[...]

[vovimayhem]

Just trying to give some potential clues.
I don't know if this is related, but i can't get pass 150 concurrent connections when binding to unix sockets, IT doesn't matter how quick or slow the ruby app is...

Nginx keeps reporting "Resource temporarily unavailable" on the 151-th concurrent connection.

This is happening to me on a CentOS server, with jRuby.

However, running the same setup on MacOS X (my dev machine) it Gets over 2500 concurrent connections.

I Hope this helps, I'm pretty stuck right now...

[wijet]

I have similar problem when using puma with haproxy (1.4.23, on 1.4.15 everything works fine). Haproxy heartbeats make puma leaks sockets.

Netstat shows that connections are in CLOSE_WAIT state and with time puma uses all available sockets and stops responding to request.

I've tested haproxy 1.4.23 with thin server and it works fine, so this seems to be puma problem.

@evanphx I was not able to reproduce this issue with sample ruby client code, but I can provide haproxy configuration (basic one), which reproduces this problem in puma.

[evanphx]

Yeah, please give me the haproxy config and I'll take a look.

[wijet]

Here is test configuration https://gist.github.com/wijet/5643212

What I noticed is that puma stops responding after threads pool runs out. If you set it to 200 for example, you will wait longer till you encounter the problem. With standard 16 threads it takes less than a minute, but even after that point, numbers of file descriptors in /proc/PUMA_PID/fd keeps rising.

If you use homebrew, here is formula for haproxy 1.4.23 https://gist.github.com/wijet/d3e691f76da6c8783b3b/raw/770ed3cef9ab93151da757408fb28a7d8044018b/haproxy.rb

[wijet]

@evanphx Thank you!