Contracts can't be written for callbacks that receive cb_kwargs

[jvani]

Description

Contracts can't be written for callbacks that receive cb_kwargs

Steps to Reproduce

1. Run scrapy check with the following spider:

class CbKwargsContract(scrapy.Spider):
    name = 'cb_kwargs_contract'

    def start_requests(self):
        yield scrapy.Request("https://httpbin.org/get", cb_kwargs={"arg1": "foo"})

    def parse(self, response, arg1):
        """
        @url https://httpbin.org/get
        """
        self.logger.info(arg1)

Expected behavior:
Ideally would be possible to add args to the contract (e.g., @url https://httpbin.org/get foo)

Actual behavior: Fails with a TypeError

Reproduces how often: 100%

Versions

Scrapy 1.7.3

[elacuesta]

I can reproduce:

E
======================================================================
ERROR: [cb_kwargs_contract] parse (callback)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../lib/python3.6/site-packages/scrapy/contracts/__init__.py", line 95, in cb_wrapper
    output = cb(response)
TypeError: parse() missing 1 required positional argument: 'arg1'

----------------------------------------------------------------------
Ran 0 contracts in 0.984s

FAILED (errors=1)

[elacuesta]

My first thought was to replace output = cb(response) with output = cb(response, **request.cb_kwargs) in https://github.com/scrapy/scrapy/blob/1.7.3/scrapy/contracts/__init__.py#L95. However, with that change I still get TypeError: parse() missing 1 required positional argument: 'arg1'.

I suspect this is due to the fact that the parsing method is not called as a Deferred callback, and the arguments are interpreted as positional instead of keyword ones. Refactoring the parsing method signature to add a default value for the expected argument (something like arg1=None) does work in that scenario, but I think we need a slightly more complex solution is in order to prevent that restriction.

Scratch that, the request is generated from the contract definition, hence the cb_kwargs attribute is in fact empty.