Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNSCACHE_ENABLED=False not working #2811

Closed
redapple opened this issue Jul 3, 2017 · 1 comment
Closed

DNSCACHE_ENABLED=False not working #2811

redapple opened this issue Jul 3, 2017 · 1 comment
Labels

Comments

@redapple
Copy link
Contributor

redapple commented Jul 3, 2017

Originally reported by @softwarevamp on StackOverflow:

When i run scrapy shell with DNSCACHE_ENABLED=False got

KeyError: 'dictionary is empty'
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: www.mydomain.com.
    2017-07-03 03:09:12 [twisted] CRITICAL: while looking up www.mydomain.com with <scrapy.resolver.CachingThreadedResolver object at 0x3fd0050>
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/lib64/python2.7/site-packages/scrapy/resolver.py", line 29, in _cache_result
        dnscache[name] = result
      File "/usr/lib64/python2.7/site-packages/scrapy/utils/datatypes.py", line 305, in __setitem__
        self.popitem(last=False)
      File "/usr/lib64/python2.7/collections.py", line 159, in popitem
        raise KeyError('dictionary is empty')
    KeyError: 'dictionary is empty'
    2017-07-03 03:09:12 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET //www.mydomain.com/> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.mydomain.com.
    Traceback (most recent call last):
      File "/usr/bin/scrapy", line 11, in <module>
        sys.exit(execute())
      File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 149, in execute
        _run_print_help(parser, _run_command, cmd, args, opts)
      File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
        func(*a, **kw)
      File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 156, in _run_command
        cmd.run(args, opts)
      File "/usr/lib64/python2.7/site-packages/scrapy/commands/shell.py", line 73, in run
        shell.start(url=url, redirect=not opts.no_redirect)
      File "/usr/lib64/python2.7/site-packages/scrapy/shell.py", line 48, in start
        self.fetch(url, spider, redirect=redirect)
      File "/usr/lib64/python2.7/site-packages/scrapy/shell.py", line 115, in fetch
        reactor, self._schedule, request, spider)
      File "/usr/lib64/python2.7/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
        result.raiseException()
      File "<string>", line 2, in raiseException
    twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: www.mydomain.com.

Any thoughts welcome

@redapple redapple added the bug label Jul 3, 2017
@redapple
Copy link
Contributor Author

redapple commented Jul 3, 2017

This patch seems to do the trick:

$ git diff
diff --git a/scrapy/resolver.py b/scrapy/resolver.py
index 4f4f0b0..0aaced7 100644
--- a/scrapy/resolver.py
+++ b/scrapy/resolver.py
@@ -22,7 +22,8 @@ class CachingThreadedResolver(ThreadedResolver):
         # to enforce Scrapy's DNS_TIMEOUT setting's value
         timeout = (self.timeout,)
         d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
-        d.addCallback(self._cache_result, name)
+        if dnscache.limit:
+            d.addCallback(self._cache_result, name)
         return d
 
     def _cache_result(self, result, name):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant