Skip to content

CachedResolver #626

Open
wants to merge 2 commits into from

2 participants

@gabtremblay

Added a new resolver that avoids to hit socket.getaddrinfo for each request.

This resolver adds a fair speed increase in Single-Host multi-request code where the dns server is slow or rate-limited. It also gives a noticeable performance increase and network noise decrease when doing 1k+ requests on the same host.

Cache timeout can be controlled with expire_minutes constructor params.

I had to fix the netutil_test.py since socket.getaddrinfo returns IPPROTO_TCP as the protocol on linux and IPPROTO_IP under windows. Tested both platforms.

Supplementary notes: If you are to compare the speed of both resolver, don't forget to add force_instance=True to your constructor since tornado will re-use the previous instance.

@gabtremblay

Sorry for the broken commit that made his way to this request yesterday,

@bdarnell
tornadoweb member

Shouldn't the system resolver be doing this kind of caching for you? I don't want to introduce a TTL-unaware DNS cache, and since getaddrinfo doesn't appear to return TTL info we can't do caching correctly at this layer.

@gabtremblay

That was my first tought also but, if windows does have default dns caching, linux does not. Theres an optional service called nscd which you can install to cache some dns request, but it does not support getaddrinfos, only gethostbyname.

@bdarnell
tornadoweb member

Hmm, that's unfortunate. I've used dnsmasq before to get system-wide dns caching, but that's kind of a pain. I'm still reluctant to add caching that doesn't respect TTL, but I'll think about it (at least as an off-by-default option)

@gabtremblay

I've did some research in the chrome and firefox dev mailing list to see how they got around this. Turns out they are in the same situation. Since they can't reliably get ttl info from platform api, they keep a TTL of 1 minute for every dns entry. This minimizes the dns poisoning risk while giving good performances. I've changed the default from 60 to 1 minute to reflect their choices.

@bdarnell bdarnell added the netutil label Jul 16, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.