[MRG+1] fix a mistake in topic spider-middleware.rst

[grammy-jiang]

The return iterable of process_spider_exception should contain scrapy.http.Request, not scrapy.http.Response.

[codecov]

Codecov Report

Merging #3201 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3201   +/-   ##
=======================================
  Coverage   82.12%   82.12%           
=======================================
  Files         228      228           
  Lines        9593     9593           
  Branches     1385     1385           
=======================================
  Hits         7878     7878           
  Misses       1456     1456           
  Partials      259      259

[lopuhin]

This fix looks consistent with the docs, process_spider_exception docs say

If it returns an iterable the :meth:process_spider_output pipeline kicks in, and no other :meth:process_spider_exception will be called.

and process_spider_output docs say

:meth:process_spider_output must return an iterable of :class:~scrapy.http.Request, dict or :class:~scrapy.item.Item objects.
:param result: the result returned by the spider :type result: an iterable of :class:~scrapy.http.Request, dict or :class:~scrapy.item.Item objects

Although this is not checked in the code

[grammy-jiang]

Hi, @lopuhin

I have written several middlewares just about this method, process_spider_exception. They all work well but I have not checked the source code. The documents could be consistent between the classes of the spider and the spider middleware.

[lopuhin]

@grammy-jiang I agree - I think your fix is good and should be merged, I was just wondering what will happen if a response is returned.

[grammy-jiang]

@lopuhin Emmm... good point.

In my view, the method process_spider_output will be called when an iterable is returned from process_spider_exception, which means the thing will happen just like a response yielded from a spider.

So your question becomes what happens when a response yielded from a spider? I have no idea, no relative experience before...

[lopuhin]

Right, returning a response from a spider looks very strange indeed!

[grammy-jiang]

@lopuhin I have checked the class spidermw.py and I have not seen any result type assert here. The type of response will go through spider middleware without any exception.

So will there any type assert in engine or scheduler? I am not good enough at twisted, would you help to check? Thanks.