[MRG] Enforce DNS resolution timeout

[redapple]

Fixes #2461

As @rolando noticed, DNS_TIMEOUT setting was never actually used by Scrapy with 14<=Twisted<=16.6, because a (1, 3, 11, 45) tuple was always passed to the DNS resolver by Twisted internally:

Successfully installed twisted-14.0.2
$ scrapy fetch --headers https://www.example.com -s DNS_TIMEOUT=15
2017-01-13 16:06:33 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: scrapybot)
2017-01-13 16:06:33 [scrapy.utils.log] INFO: Overridden settings: {'DNS_TIMEOUT': '15'}
(...)
CachingThreadedResolver.getHostByName(name='www.example.com', timeout=(1, 3, 11, 45))
2017-01-13 16:06:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com> (referer: None)


Successfully installed twisted-15.5.0
$ scrapy fetch --headers https://www.example.com -s DNS_TIMEOUT=15
2017-01-13 16:07:05 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: scrapybot)
2017-01-13 16:07:05 [scrapy.utils.log] INFO: Overridden settings: {'DNS_TIMEOUT': '15'}
(...)
CachingThreadedResolver.getHostByName(name='www.example.com', timeout=(1, 3, 11, 45))
2017-01-13 16:07:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com> (referer: None)


Successfully installed twisted-16.6.0
$ scrapy fetch --headers https://www.example.com -s DNS_TIMEOUT=15
2017-01-13 16:07:20 [scrapy.utils.log] INFO: Scrapy 1.3.0 started (bot: scrapybot)
2017-01-13 16:07:20 [scrapy.utils.log] INFO: Overridden settings: {'DNS_TIMEOUT': '15'}
(...)
CachingThreadedResolver.getHostByName(name='www.example.com', timeout=(1, 3, 11, 45))
2017-01-13 16:07:21 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com> (referer: None)

And Twisted 16.7 actually stops passing this default value, but a tuple (or list) of ints is expected.

[redapple]

This is worth a backport on 1.0, 1.1 and 1.2 branches.

[codecov-io]

Codecov Report

Merging #2496 into master will increase coverage by 0.02%.

@@            Coverage Diff             @@
##           master    #2496      +/-   ##
==========================================
+ Coverage   83.46%   83.48%   +0.02%     
==========================================
  Files         161      161              
  Lines        8780     8779       -1     
  Branches     1288     1287       -1     
==========================================
+ Hits         7328     7329       +1     
+ Misses       1204     1203       -1     
+ Partials      248      247       -1

Impacted Files	Coverage Δ
scrapy/resolver.py	89.47% <100%> (+9.47%)	✅

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ca191e...d2e9ea0. Read the comment docs.

[dangra]

it means getHostByName callers can't not override timeout which is acceptable but it is possible for self.timeout to be a tuple and convert it only if not. This way we don't lost access to existent functionality provided by t.i.b.ThreadedResolver.

[redapple]

Sorry, I don't understand.

[dangra]

np, it's me being obfuscated :)

the question is: Can DNS_TIMEOUT be a tuple?

[dangra]

the question is: Can DNS_TIMEOUT be at tuple?

I guess not without some other major changes at

scrapy/scrapy/crawler.py

Line 290 in 4ca191e

timeout=self.settings.getfloat('DNS_TIMEOUT')

[dangra]

Do I understand that you would like to allow users to pass a tuple of ints/floats as DNS_TIMEOUT setting value?

Yes, that was it.

[kmike]

@redapple could you please add a comment above this line which explains why timeout argument is ignored?

[redapple]

@kmike , done.

[dangra]

LGTM.

[sibiryakov]

I would vote for accepting tuple or int/float by Scrapy. There are still desperate people on this planet who tries to do broad crawling and they could benefit from this functionality. Cutting it off, you also cut DNS retries management. Someday DNS providers will thank you for having this option.

[redapple]

@sibiryakov , I do not understand why accepting tuple helps with broad crawls (honest question).
This patch is about forcing the DNS resolution timeout to the value set in scrapy settings (prior, only Twisted's default 60s was being used).
Other changes to DNS resolution will probably be based on the new pluggable resolver interface in Twisted 17.

[redapple]

@sibiryakov , are you ok with opening a seperate new issue to discuss DNS resolution implementation and configuration options that would play nicer for broad crawls?

[sibiryakov]

sure @redapple