CrawlSpider improvements

[dangra]

A ticket to kickoff the discussion on CrawlSpider enhancements (if any)

ideas:

• Simplify rule definitions by using an implicit linkextractor instanciated from LxmlLinkExtractor.
• ... what else?

[pablohoffman]

• any good idea from SEP-014 (if any)
• support overriding parse? (Promote a new CrawlSpider that allows overriding parse #712)

[nyov]

in regards to "overriding parse", my proposal is at #732 (but a bit more generic than CrawlSpider)
In essence, I believe in decoupling the Scraper's entrypoint from the user-facing parse.

+------------------+-----------------+---------------------+
| Scraper (caller) | Spider (callee) | UserSpider(Spider)  |
+------------------+-----------------+---------------------+
| call_spider() ---> init()/_parse() |                     |
|                  | |`-> important()|                     |
|                  | `--> parse()  <-- parse() /wo super() |
+------------------+-----------------+---------------------+

Another thing I've been thinking on in the past was designing spiders around Mixin-classes instead. Maybe that doesn't belong here. But the outcome would be that Spider "ideas" are self-contained and combineable, something like MySpider(Spider, Crawl, Init, Csv).

With InitSpider this is probably already possible to some extent, but currently it's not guranteed that classes could work together.

[redapple]

I would also add something related to #929, that is passing the response to process_request in _requests_to_follow, so that one can tweak the generated requests with some context. (If there's another way currently than overriding _request_to_follow, it'd be happy for it to be documented :)

Oh, and also adding errback to rules could be handy at times: see http://stackoverflow.com/a/35870000/2572383

[guillaumedsde]

I would also add something related to #929, that is passing the response to process_request in _requests_to_follow, so that one can tweak the generated requests with some context. (If there's another way currently than overriding _request_to_follow, it'd be happy for it to be documented :)

Oh, and also adding errback to rules could be handy at times: see http://stackoverflow.com/a/35870000/2572383

is this a planned feature?

[elacuesta]

is this a planned feature?

@guillaumedsde Please see #3682 (included in version 1.7.1) and #4000 (in progress).

[elacuesta]

Seems to me like everything has been addressed 👌