Custom resolver

[psznm]

Hi, I am trying to understand the characteristics of zio and tried the new custom resolver, I have a test that repeatedly resolves single domain N times. With max_threads=1

For N = 100 my test takes ~0.02 seconds.
For N = 200 it takes 5 seconds
For N = 450 it still takes 5 seconds
For N = 500 it takes 10 seconds
For N = 1000 it takes 10 seconds
For N = 1100 i start getting NameServerFailure

Note that I run vtable.netLookup 2x per resolve (don't ask). I also tried to run lookup only once per resolve, while times stayed roughly the same with N*2, the NameServerFailures still started at N=1100 (with half the vtable.netLookup runs)

I also tried adding batching to resolve 100 at a time, which solved both the huge jumps in time to finish and the NameServerFailures

Is the issue on my part and I need to prevent too many async tasks? 400 simultaneous tasks feels like it shouldn't be too many to start causing such delays. Any insight?

[lalinsky]

Can you share the test? The thing is, there is in-memory cache, with TTL between 1 and 60 seconds, so you are repeatedly requesting the same domain. It should be already cached and return immediately.

Also keep in mind, that the resolver is very new. I only tested it with my local SystemD DNS server. I possibly also need to add more logging to make it clear where is the error coming from.

[lalinsky]

Ok, I've tried it myself and can reproduce the delays, it's very strange, will dig into that.

[lalinsky]

In my case, the delay seems to be coming from the DNS server itself. Requests timeout after it accumulates some number of pending queries. Then they clear and things are going again. The 5s is the timeout from resolv.conf, you most likely have 2 retries configured, so it can delay up to 10 seconds and then it fails.

I can see two approaches:

1. coalesce requests for the same domain, if we already have one query in flight, just make the other tasks wait for it. this only helps with the same domain requests, but maybe that's common enough.

2. for the case of e.g. 1000 different domains, maybe I need to do some rate limiting, to avoid hitting the timeouts.

[psznm]

I see. I also use systemd resolver on my PC, so I went to try on my development container which runs bind resolver. And I've run into another issue. Seems like missing /etc/resolv.conf is a problem, which shouldn't be and resolver should try default 127.0.0.1:53

Anyway, after adding /etc/resolv.conf, I've gotten another surprising result. Resolving 1000 domains (unbatched) no longer took 10 seconds, but only 0.16s with the bind resolver. So it seems it must be systemd resolver issue.

The NameServerFailure seems like it might be zio issue though. It starts happening at exactly 1020 resolves with both resolvers.

• 1019 works fine every single time.
• 1020 gives me 1x NameServerFailure, around 50:50 times.
• 1030 gives me 11x NameServerFailure, around 50:50 times.
• 1100 rarely works without NameServerFailure.
• 1200 gives NameServerFailures every single time

Also note that with systemd resolver it was easy to notice that I got the NameServerFailure immediately, seemingly before other queries even started resolving. Feels like some buffer/queue could be getting full and this is a symptom. Maybe this is fine, as my test is very contrived (firing off thousands of queries at once).

[lalinsky]

The NameServerFailure seems like it might be zio issue though. It starts happening at exactly 1020 resolves with both resolvers.

This could be fd limit issue, check ulimit -n. Currently, each query needs a separate UDP socket, and with the default 1024 limit, you could easily reach it. See if extending the limit helps.

Maybe I should do a pool of sockets, limited to some number and share them. That would be faster and easier to scale, but at the same time, it needs more runtime coordination and management.

[lalinsky]

Yeah, the shared socket pool is what c-ares is doing, with a configurable number of active queries per socket.

[lalinsky]

And interestingly, Go's resolver also does one socket per query, much like I decided to do, so I guess it's not a horrible choice. :)

[psznm]

Yeah, it was just the contrived test. The actual service that my resolver is used in has pretty much unlimited sockets and using bind resolver. So all will be behaving great even if loads as in the tests would be reached, which is extremely unlikely.

Thanks alot. I've submitted PR for the missing resolv.conf handling #411