Support for async callbacks

[wRAR]

This PR adds support for defining async def parse_* and using both yield and await inside, with or without the asyncio reactor. There is one caveat: if the asyncio reactor is enabled, you cannot await on Deferreds directly and need to wrap them. I've added this to docs.

This change was tested on two real-world projects, for one of them I also replaced using inline_requests to get HTTP responses inside a callback with alternatively treq or aiohttp. I didn't test the performance, but as far as I can see the existing non-async callbacks shouldn't have any significant code path changes so there should be no regressions and if there are any perf problems with async callbacks we can solve that later.

This doesn't require or support async def start_requests.

My main concern with this is large and non-intuitive scrapy.utils.defer._AsyncCooperatorAdapter. I would like to replace it with something better if that's possible but for now it seems to work correctly. I've tried to document it as comprehensively as possible.

I've split it in several commits with different features in case this help with review for someone.

[codecov]

Codecov Report

Merging #4978 (8659bf4) into master (737f396) will increase coverage by 0.14%.
The diff coverage is 98.44%.

❗ Current head 8659bf4 differs from pull request most recent head c7b90c6. Consider uploading reports for the commit c7b90c6 to get more accurate results

@@            Coverage Diff             @@
##           master    #4978      +/-   ##
==========================================
+ Coverage   88.70%   88.85%   +0.14%     
==========================================
  Files         162      162              
  Lines       10787    10962     +175     
  Branches     1851     1894      +43     
==========================================
+ Hits         9569     9740     +171     
  Misses        942      942              
- Partials      276      280       +4     

Impacted Files	Coverage Δ
scrapy/spidermiddlewares/urllength.py	88.00% <90.90%> (-2.48%)	⬇️
scrapy/spidermiddlewares/offsite.py	98.41% <94.11%> (-1.59%)	⬇️
scrapy/spidermiddlewares/depth.py	97.67% <95.45%> (-2.33%)	⬇️
scrapy/core/spidermw.py	99.44% <99.08%> (-0.56%)	⬇️
scrapy/core/scraper.py	85.79% <100.00%> (+0.24%)	⬆️
scrapy/middleware.py	92.85% <100.00%> (ø)
scrapy/spidermiddlewares/referer.py	93.24% <100.00%> (+0.13%)	⬆️
scrapy/utils/asyncgen.py	100.00% <100.00%> (ø)
scrapy/utils/defer.py	97.29% <100.00%> (+1.33%)	⬆️
scrapy/utils/python.py	88.58% <100.00%> (+0.93%)	⬆️
... and 2 more

[wRAR]

Not sure why some related tests fail on 3.6 asyncio-pinned, I cannot reproduce this locally at this time.

[wRAR]

Now I can reproduce them on a freshly built 3.6.12 while it worked on my old 3.6.9, both from pyenv.

[wRAR]

It's Twisted 17.9.0-specific again.

[wRAR]

pytest-twisted installs the asyncio reactor with eventloop=None, so on Twisted 17.9.0 uses new_event_loop(). We added a workaround for this in Scrapy itself, in #4841 / #4872, but it's much harder to do for pytest-twisted, unless they add some support for that (or we drop 17.9.0). It's probably possible to install the reactor in conftest.py though.

[Gallaecio]

Thanks for the separate commits, it definitely made review much more approachable.

I think it may be best to add tests for parallel_async that ensure complete code coverage for _AsyncCooperatorAdapter, specially given its complexity (I won’t claim to fully understand it).

I love the documentation entry on how to handle waiting on deferreds now! When releasing 2.5, we should probably remember to add a brief mention about this to the release notes as a backward-incompatible change (on an experimental feature, but still), linking to the section for further details.

[Gallaecio]

I am proposing some changes documentation-wise, but otherwise this looks great to me.

[kmike]

Looks great @wRAR! +1 to merge, after merging in a clean up by @Gallaecio.

I only got superficial understanding of how the code works, but what I understood makes total sense to me :) Tests also makes sense, and the overall approach looks right.

[auxsvr]

Shouldn't we also update CrawlSpider._parse_response, CrawlSpider._requests_to_follow, LxmlParserLinkExtractor._extract_links etc to support coroutines and async generators?

[wRAR]

@auxsvr on one hand we haven't thought about this and doing this shouldn't be necessary to merge this feature as existing CrawlSpider-based spiders should continue to work and this support can be added separately (if you suspect this is not true please voice your suspicions), on other hand it should be possible to convert any CrawlSpider-based spider to a normal one if one wants to use async callbacks.

[kmike]

A fix for the typing issues: wRAR#2 (and wRAR@56e2eea specifically)

[Gallaecio]

Overwhelmingly amazing!

[kmike]

Tremendous work, thanks @wRAR!