DNS_Mgr: Fixes around timeouts and IO loop behavior

[awelzel]

On Slack we were discussing below reproducer causing memory growth due to un-fulfilled when statements related to DNS_Mgr, lookup_addr(), ...

The two commits in this PR fix the state growth.

The most problematic scenario is triggered when 20 (MAX_PENDING_REQUESTS) DNS requests are pending, but none of them is ever responded to (so there's no FD activity in the future). Due to essentially non-existing timeout functionality, this would cause any future lookup_addr() calls to never be resolved.

@timwoj , I think this is 6.0.1 material, minimally 6.0.2 if that ship has sailed. Unfortunately I'm uncertain how this is best tested automatically without putting in much effort.

---

Think I have a better one because possibly a bit more real (DNS server restarting) - seems there's more going on.

• Replace 192.168.0.1 above with 10.0.0.1, this results in NXdomain responses every 5 seconds on my system. In-between the cache is used. No memory growth.
  Stop local resolver: systemctl stop systemd-resolved
  Wait until cache timeout expires, observe memory growth even after 5 second DNS timeout should kick in.
  Restart resolver: Expect memory growth to stop because DNS is available again. It doesn't recover and memory keeps growing.

Sketch of script for reproduction:

redef exit_only_after_terminate=T;

global s = F;

global x: table[addr] of string;

event stats_tick() {
        local stats = get_proc_stats();
        print "stats", network_time(), stats$mem, get_dns_stats();
        schedule 1sec { stats_tick() };  # default DNS timeout
}

function f(a: addr) {
        when [a] ( local name = lookup_addr(a) )
                {
                x[a] = cat_sep(" ", "", network_time(), a, name);
                }
}

event tick() {
        f(10.0.0.1);
        schedule 10msec { tick() };
}

event zeek_init()
        {
        event tick();
        event stats_tick();
        }

[timwoj]

I agree this is a good idea for 6.0.1

[zeek-bot]

This pull request has been mentioned on Zeek. There might be relevant details there:

https://community.zeek.org/t/zeek-is-consuming-100-ram-memory/7128/4