dnsdist: Too many open files when backend authoritative server restarts

[rodecker]

Hello,

We have dnsdist running in front of a single gdnsd instance. We also have carbonServer configured to send data to a Graphite instance. Occasionally when gdnsd restarts (for example because of a configuration update) we see the following:

Jan 25 01:34:26 hostname dnsdist[17920]: Marking downstream 127.0.0.1:5300 as 'down'
Jan 25 01:34:27 hostname dnsdist[17920]: Marking downstream 127.0.0.1:5300 as 'up'
Jan 25 01:49:28 hostname dnsdist[17920]: Marking downstream 127.0.0.1:5300 as 'down'
Jan 25 01:49:58 hostname dnsdist[17920]: Problem sending carbon data: Too many open files
Jan 25 01:50:47 hostname dnsdist[17920]: While reading a TCP question: accepting new connection on socket: Too many open files
Jan 25 01:50:58 hostname dnsdist[17920]: Problem sending carbon data: Too many open files
Jan 25 01:52:58 hostname dnsdist[17920]: message repeated 2 times: [ Problem sending carbon data: Too many open files]
Jan 25 01:53:57 hostname dnsdist[17920]: While reading a TCP question: accepting new connection on socket: Too many open files

Even though gdnsd is available again, dnsdist does no longer see it as up. A restart of dnsdist is required.

$ dnsdist -V
dnsdist 1.0.0-alpha1

$ cat /etc/dnsdist/config
controlSocket("0.0.0.0")
webserver("0.0.0.0:8080", "")
setKey("")
setACL({"0.0.0.0/0", "::/0"})
carbonServer('', '', 60)
truncateTC(true) -- fix up possibly badly truncated answers from pdns 2.9.22

warnlog(string.format("Script starting %s", "up!"))

-- define the good servers
newServer {address="127.0.0.1:5300", useClientSubnet=true}

$ cat /proc/sys/fs/file-max
900000

[ahupowerdns]

do you have the fd-usage metric available somewhere?

[rodecker]

[rgacogne]

Based on a PCAP file kindly provided by @rodecker, dnsdist accepts a new TCP connection which is closed right away by the remote end. dnsdist acknowledges the FIN, but sometimes does not close the connection (no FIN or RST sent). I still trying to figure that out.

[rgacogne]

So apparently the TCP acceptor thread accepts a new connection then get a file descriptor in order to notify the selected TCP client thread. For a reason yet unknown, the value of this FD is 0, which is mapped to /dev/null, causing the connection to be leaked:

[pid 25209] <... accept resumed> {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("192.0.2.1")}, [16]) = 85
[pid 25209] write(0, "\240\23\0,\222\177\0\0", 8) = 8

lrwx------ 1 root root 64 Jan 26 14:32 0 -> /dev/null
lrwx------ 1 root root 64 Jan 26 14:32 1 -> /dev/null

The above-mentioned commit adds several checks and minor fixes to try to fix this bug, but I am not 100% convinced it will be enough, as I am not able to reproduce it.

[rodecker]

Thanks! This appears to have solved it. I have not seen any more occurrences of this issue as of yet.

[rgacogne]

@rodecker Thanks for the update!

[dimsua]

dnsdist-1.0.0-1pdns.el6.x86_64

Apr 27 13:00:06 client5545 dnsdist[6911]: Read configuration from '/etc/dnsdist/dnsdist.conf'
Apr 27 13:00:06 client5545 dnsdist[25110]: Got control connection from 127.0.0.1:49168
Apr 27 13:00:06 client5545 dnsdist[25110]: Closed control connection from 127.0.0.1:49168
Apr 27 13:00:07 client5545 dnsdist[25110]: While reading a TCP question: accepting new connection on socket: Too many open files
Apr 27 13:00:07 client5545 dnsdist[25110]: While reading a TCP question: accepting new connection on socket: Too many open files
Apr 27 13:00:20 client5545 dnsdist[6929]: Read configuration from '/etc/dnsdist/dnsdist.conf'
Apr 27 13:00:20 client5545 dnsdist[6929]: Fatal error: connecting socket to 127.0.0.1:5199: Connection refused

[rgacogne]

Hi dimsua,

Thank you for reporting this, do you have any additional information you could provide? Do you export metrics to a graphite/metronome server?

[dimsua]

We do not use graphite/metronom, use only zabbix. we write simple plugin for zabbix
collect queries, example:
echo "dumpStats()"|dnsdist --client|grep queries|awk '{print $3,$4}'|grep "^queries "|awk '{print $2}'
we need monitoring fd-usage?