Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError in is_generator_with_return_value #4597

wRAR opened this issue May 27, 2020 · 0 comments · Fixed by #4599

KeyError in is_generator_with_return_value #4597

wRAR opened this issue May 27, 2020 · 0 comments · Fixed by #4599


Copy link

wRAR commented May 27, 2020

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/app/python/lib/python3.8/site-packages/scrapy/core/downloader/", line 42, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "/app/python/lib/python3.8/site-packages/scrapy/core/", line 60, in process_spider_input
    return scrape_func(response, request, spider)
  File "/app/python/lib/python3.8/site-packages/scrapy/core/", line 148, in call_spider
    warn_on_generator_with_return_value(spider, callback)
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/", line 202, in warn_on_generator_with_return_value
    if is_generator_with_return_value(callable):
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/", line 180, in is_generator_with_return_value
    return _generator_callbacks_cache[callable]
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/", line 281, in __getitem__
    return super(LocalWeakReferencedCache, self).__getitem__(key)
  File "/usr/local/lib/python3.8/", line 383, in __getitem__
KeyError: <weakref at 0x7f06ff011720; to 'method' at 0x7f07042b5e00 (parse_foo)>

This is Scrapy 2.0.1. The problem happens only sometimes, but in different spiders in the same project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging a pull request may close this issue.

1 participant