TypeError module, class, method, function, traceback, frame, or code object was expected, got partial #5592
Labels
Comments
|
You shouldn't use partials as callbacks. It used to be the case that it could indirectly cause serialization issues (see #1138 (comment)), this exception is different because the code is newer but the point remains. Use Request.cb_kwargs instead to pass arguments across. |
|
@elacuesta @kmike |
|
I think it may be worth it failing more gracefully on partial functions. |
|
My monky path: def _manky_path_is_generator_with_return_value():
import ast
import inspect
from functools import partial
import re
import scrapy.utils.misc as pathed
_generator_callbacks_cache = pathed._generator_callbacks_cache
walk_callable = pathed.walk_callable
def is_generator_with_return_value(callable):
"""
Returns True if a callable is a generator function which includes a
'return' statement with a value different than None, False otherwise
"""
if callable in _generator_callbacks_cache:
return _generator_callbacks_cache[callable]
def returns_none(return_node):
value = return_node.value
return (
value is None or (
isinstance(value, ast.NameConstant) and value.value is None))
if inspect.isgeneratorfunction(callable):
func = callable
while isinstance(func, partial):
func = func.func
code = re.sub(r"^[\t ]+", "", inspect.getsource(func)) # callable))
tree = ast.parse(code)
for node in walk_callable(tree):
if isinstance(node, ast.Return) and not returns_none(node):
_generator_callbacks_cache[callable] = True
return _generator_callbacks_cache[callable]
_generator_callbacks_cache[callable] = False
return _generator_callbacks_cache[callable]
pathed.is_generator_with_return_value = is_generator_with_return_value |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Error when responce calbeck is patial
Steps to Reproduce
Expected behavior: [What you expect to happen]
Run callback
Actual behavior: [What actually happens]
Error
Reproduces how often: [What percentage of the time does it reproduce?]
Always
Versions
Please paste here the output of executing
scrapy version --verbosein the command line.Additional context
Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use GitHub Flavored Markdown to make the text look better.
The text was updated successfully, but these errors were encountered: