CachedResolver #626

Open
wants to merge 2 commits into
from

Conversation

Projects
None yet
2 participants
@gabtremblay

Added a new resolver that avoids to hit socket.getaddrinfo for each request.

This resolver adds a fair speed increase in Single-Host multi-request code where the dns server is slow or rate-limited. It also gives a noticeable performance increase and network noise decrease when doing 1k+ requests on the same host.

Cache timeout can be controlled with expire_minutes constructor params.

I had to fix the netutil_test.py since socket.getaddrinfo returns IPPROTO_TCP as the protocol on linux and IPPROTO_IP under windows. Tested both platforms.

Supplementary notes: If you are to compare the speed of both resolver, don't forget to add force_instance=True to your constructor since tornado will re-use the previous instance.

@gabtremblay

This comment has been minimized.

Show comment
Hide comment
@gabtremblay

gabtremblay Nov 11, 2012

Sorry for the broken commit that made his way to this request yesterday,

Sorry for the broken commit that made his way to this request yesterday,

@bdarnell

This comment has been minimized.

Show comment
Hide comment
@bdarnell

bdarnell Nov 11, 2012

Member

Shouldn't the system resolver be doing this kind of caching for you? I don't want to introduce a TTL-unaware DNS cache, and since getaddrinfo doesn't appear to return TTL info we can't do caching correctly at this layer.

Member

bdarnell commented Nov 11, 2012

Shouldn't the system resolver be doing this kind of caching for you? I don't want to introduce a TTL-unaware DNS cache, and since getaddrinfo doesn't appear to return TTL info we can't do caching correctly at this layer.

@gabtremblay

This comment has been minimized.

Show comment
Hide comment
@gabtremblay

gabtremblay Nov 11, 2012

That was my first tought also but, if windows does have default dns caching, linux does not. Theres an optional service called nscd which you can install to cache some dns request, but it does not support getaddrinfos, only gethostbyname.

That was my first tought also but, if windows does have default dns caching, linux does not. Theres an optional service called nscd which you can install to cache some dns request, but it does not support getaddrinfos, only gethostbyname.

@bdarnell

This comment has been minimized.

Show comment
Hide comment
@bdarnell

bdarnell Nov 12, 2012

Member

Hmm, that's unfortunate. I've used dnsmasq before to get system-wide dns caching, but that's kind of a pain. I'm still reluctant to add caching that doesn't respect TTL, but I'll think about it (at least as an off-by-default option)

Member

bdarnell commented Nov 12, 2012

Hmm, that's unfortunate. I've used dnsmasq before to get system-wide dns caching, but that's kind of a pain. I'm still reluctant to add caching that doesn't respect TTL, but I'll think about it (at least as an off-by-default option)

@gabtremblay

This comment has been minimized.

Show comment
Hide comment
@gabtremblay

gabtremblay Nov 12, 2012

I've did some research in the chrome and firefox dev mailing list to see how they got around this. Turns out they are in the same situation. Since they can't reliably get ttl info from platform api, they keep a TTL of 1 minute for every dns entry. This minimizes the dns poisoning risk while giving good performances. I've changed the default from 60 to 1 minute to reflect their choices.

I've did some research in the chrome and firefox dev mailing list to see how they got around this. Turns out they are in the same situation. Since they can't reliably get ttl info from platform api, they keep a TTL of 1 minute for every dns entry. This minimizes the dns poisoning risk while giving good performances. I've changed the default from 60 to 1 minute to reflect their choices.

@bdarnell bdarnell added the netutil label Jul 16, 2014

@pfreixes pfreixes referenced this pull request in aio-libs/aiohttp Apr 16, 2017

Merged

Add new ttl option to expire DNS entries after N seconds #1819

3 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment