CallbackKeywordArgumentsContract

[elacuesta]

Fixes #3985

This patch works partially, in the sense that it prevents the reported TypeError (I checked, for instance by adding a simple print(request.cb_kwargs) to https://github.com/scrapy/scrapy/blob/1.7.3/scrapy/core/scheduler.py#L90, and the generated request does include a valid cb_kwargs attribute). However, adding the @cb_kwargs contract causes the whole callback to be excluded from the tests, and I can't for the life of me figure out why.

Update: Working now, thanks to @victor-torres 🚀

Remaining tasks:

• Docs
• Tests
• Figure out why it doesn't work

[codecov]

Codecov Report

Merging #3988 into master will decrease coverage by 0.01%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master    #3988      +/-   ##
==========================================
- Coverage   85.39%   85.38%   -0.02%     
==========================================
  Files         167      167              
  Lines        9724     9730       +6     
  Branches     1456     1456              
==========================================
+ Hits         8304     8308       +4     
- Misses       1162     1164       +2     
  Partials      258      258

Impacted Files	Coverage Δ
scrapy/settings/default_settings.py	98.7% <ø> (ø)	⬆️
scrapy/contracts/__init__.py	81.3% <66.66%> (ø)	⬆️
scrapy/contracts/default.py	84% <66.66%> (-2.37%)	⬇️

[victor-torres]

Since you're wrapping the callback, you need to receive the kwargs here as well.

@wraps(cb)
def cb_wrapper(response, **cb_kwargs):
    pass

[victor-torres]

You also need to do it in the other wrappers.

[victor-torres]

Then, instead of using the synthetic request's kwargs, you need to pass the wrapped kwargs to the callback.

try:
    output = cb(response, **cb_kwargs)
except Exception:
    ...

[victor-torres]

Despite of that, you might have some problems when trying to pass a Python object through the cb_kwargs because it won't be possible to serialize and deserialize data.

[elacuesta]

That's true, but it's a current limitation of Request.meta as well, and it's solved by enqueuing to memory instead of disk. I don't think we need to worry about that in the context of Scrapy contracts though, since the requests are built from docstings and the cb_kwargs attribute in particular is decoded as JSON.

[kmike]

Thanks @elacuesta, and thanks @victor-torres and @Gallaecio for reviews!