How does uvloop (or I think libuv) does dns resolution, is it fully async?

[santagada]

I've read the libuv documentation and it appears that it is, and that uvloop is calling directly into it. So if I use aiohttp to do requests will the dns resolution be async as well?

And do you know why node still uses c-ares to do dns resolution if libuv is fully async?

[1st1]

Yes, libuv uses a thread pool to do getaddrinfo and getnameinfo, and so does uvloop. AFAIK nodejs doesn't use c-ares anymore.

I'm still deciding how to approach this problem -- should we use c-ares? or getdns? Or keep things as is, since DNS in a thread pool is fine for most use cases?

[infra-0-0]

somewhat relevant to the conversation: https://gist.github.com/deltaindiatango/7bf157202c17681d62667a006eb15abd

found some existing pure-Python resolver, did basic work to make async.

It's not noticably slower than getaddrinfo/getnameinfo.

[benjamingr]

Hi, Node guy here - Node.js does use c-ares. Here's discussion about replacing it nodejs/node#1013

DNS thread pool is a compromise. Node documents .lookup:

Though the call to dns.lookup() will be asynchronous from JavaScript's perspective, it is implemented as a synchronous call to getaddrinfo(3) that runs on libuv's threadpool. Because libuv's threadpool has a fixed size, it means that if for whatever reason the call to getaddrinfo(3) takes a long time, other operations that could run on libuv's threadpool (such as filesystem operations) will experience degraded performance. In order to mitigate this issue, one potential solution is to increase the size of libuv's threadpool by setting the 'UV_THREADPOOL_SIZE' environment variable to a value greater than 4 (its current default value). For more information on libuv's threadpool, see the official libuv documentation.

And .resolve:

These functions are implemented quite differently than dns.lookup(). They do not use getaddrinfo(3) and they always perform a DNS query on the network. This network communication is always done asynchronously, and does not use libuv's threadpool.

[1st1]

@benjamingr Benjamin, thanks for chiming in! My current understanding is that for asyncio we'll have to approach this problem similarly to Node: add a completely new set of DNS APIs that doesn't use getaddrinfo and uses c-ares or getdns instead. Maybe we want to make it possible to plug custom DNS resolvers so that create_connection can use them.

[benjamingr]

@1st1 I'm in the opinion you're doing an excellent job using libuv and that you shouldn't worry about this until it becomes a real problem.

I'd argue that the vast majority of programs don't perform a huge amount of uncacheable DNS queries and that it's an edge case. I'd invest more in getting more library adoption and feedback.

In practice the wrapper will behave like users expect for the vast majority of users. I don't see how you'd solve this without using something other than libuv either.

[asvetlov]

aiohttp 1.0 uses c-ares by default if aiodns is installed.

[WGH-]

But won't c-ares break expectations in case /etc/nsswitch.conf has somethings more complex than hosts: files dns, like mdns or resolved (systemd-resolved-based plugin, which supports LLMNR, per-interface domain names, etc.)?

[asvetlov]

@WGH-
Honestly I don't know how these tools are widespread.
Let's look on users feedback.

[DeoLeung]

we encountered a dns problem using uvloop with tornado

socket.gaierror: [Errno -3] Temporary failure in name resolution

this hangs our program

maybe there's a way to set a timeout on the lookup or may uvloop dns lookup async?

---

I found UV_THREADPOOL_SIZE may help, so I'm increasing it to maximum 128, still I think getting a way to timeout dns lookup or throw error will help the program, seems hanging is really bad...

[yxy]

we encountered a dns problem using uvloop with tornado

socket.gaierror: [Errno -3] Temporary failure in name resolution

this hangs our program

maybe there's a way to set a timeout on the lookup or may uvloop dns lookup async?

I found UV_THREADPOOL_SIZE may help, so I'm increasing it to maximum 128, still I think getting a way to timeout dns lookup or throw error will help the program, seems hanging is really bad...

same issue here. +1