Experiencing network slowdowns when using CoreDNS to resolve hosts file.

[jmuchovej]

I put together a container that uses CoreDNS to resolve ZeroTier peers ahead of all other domains. However, as someone not at all versed in DNS architecting/etc. I believe I've made some mistakes that are the cause of network slowdowns.

Background

Here's the repository: https://github.com/jmuchovej/zerodns

Specifically, here's a copy-paste of the Corefile ZeroDNS uses:

.:5353 {
    errors
    log
    hosts /config/hosts {
        reload 15m
        fallthrough
    }
    forward . /etc/resolv.conf
    reload 15m
}

ZeroDNS itself just generates the /config/hosts file. Each IP address has at least 6 aliases (and I've made sure to have each IP address occur only once in /config/hosts).

In my own ZeroDNS setup, I have about 24 entries in /config/hosts but have noticed that when I set ZeroDNS to act as my [only] DNS (with fallbacks of 1.1.1.1 and 1.0.0.1 provided through docker ... --dns 127.0.0.1 --dns 1.1.1.1 --dns 1.0.0.1) many websites resolve very slowly.

Something that might be worth noting is that /etc/resolv.conf might not be adding --dns entries. 😕

Whenever I attempt to enter a running instance of ZeroDNS, dumping the contents of /etc/resolv.conf shows me default values like:

nameserver 127.0.0.11
options ndots:0

While this might be a question for the folks at docker – is this something that could be contributing to the slow forwarding/evaluation? (Though if that's the case, I'm not sure how domains are being resolved unless there are "default records" that my laptop (MacOS) falls back to in the absolute worst case – which makes sense given clearing all custom DNS entries immediately reverts to the values provided by my router.)

Questions

1. In an effort to maintain code simplicity, I opted for /etc/hosts, but I imagine there's some trade-off in this simplicity. Are alternative lookup formats more performant? (If so, is there a minimum number of addresses/names at which they actually demonstrate said performance gains?)
2. Would having a more specific Corefile likely improve performance? (e.g. see Corefile below.)

   .network1.zt:5353 {  # or even: `.network1.zt:5353 .network2.zt:5353 {`
           hosts /config/hosts {
           reload 15m
       }
   }
   
   .network2.zt:5353 {
       hosts /config/hosts {
           reload 15m
       }
   }
   
   .:5353 {
       errors
       log
       forward . /etc/resolv.conf
       reload 15m
   }
   

3. Are there broad suggestions on how to ensure CoreDNS runs as quickly as possible?

[chrisohaver]

Add the loop plugin to the corefile to see if you have a forwarding loop. With loopback addresses like 127.0.0.1 in /etc/resolv.conf, a self forwarding loop is a distinct possibility. I see that CoreDNS listens on 5353, and forwards on 53, but not sure if docker container port mapping is redirecting 53 to 5353 to make a forwarding loop.

[jmuchovej]

Thanks for the quick response! 🙂

So, I added loop to the Corefile:

.:5353 {
    errors
    log
    hosts /config/hosts {
        reload 15m
        fallthrough
    }
    forward . /etc/resolv.conf
    reload 15m
}

I've been browsing incessantly for the last 20 minutes and docker compose logs zerodns | grep -i loop dumps nothing. (Which is good.)

After looking into the Docker "DNS" a bit more, it seems like the --dns ... options passed to the container are used to modify something related to Docker's SDN. What I've synthesized from a few articles/SO posts is that all containers (at least on MacOS) point to something like 127.0.0.11 which Docker seems to manage (though ifconfig doesn't show any devices using 127.0.0.11), the Docker forwards responses to said DNSes.

Based on my own understanding of how this ends up working, if I query github.com, that will be sent to the container (then to CoreDNS), who fails to find github.com in /config/hosts so it then forwards to 127.0.0.11 (which somehow Docker gets a hold of), then github.com gets forwarded to the DNSes (127.0.0.1, 1.1.1.1, and 1.0.0.1). Presumably, this would create the very loop you mentioned (yet, somehow it doesn't – weirdly, if I don't add 127.0.0.1 as a DNS I can't access anything).

(Also, I should note that some of my slowdowns appear to be due to service outages, so maybe I was jumping the gun. 😅)

[chrisohaver]

Cool - If loop detects a loop, it will do so somewhat immediately after starting up, log an error and shut down. IIRC loop stops checking after some short period of time (less than a minute).

[jmuchovej]

Sweet.

Also, if you have time/etc. I'd appreciate some responses/feedback on the questions in my OP. 🙂

[chrisohaver]

In response the the OP questions...

If all of your hosts fall into known predictable zones (and nothing else is in those zones), then you could use the file plugin. I suspect the file plugin is more performant than hosts (map vs tree) - but I don't know for sure, and it might not be for a short list of hosts. Hosts is much easier to use than file, the latter requiring a significantly more verbose dns zone file.
Even if you stick with the hosts plugin, the Corefile in 2 would theoretically help, since it would only check hosts for those zones. I say theoretically, because if the list of hosts is short, then it might not amount to a significant portion of the entire DNS transaction.