Non-blocking autologin

[lopuhin]

I make requests to autologin via normal scrapy requests and process responses in callbacks. Requests before login and during logout are queued and scheduled after login.

[codecov-io]

Current coverage is 79.44%

Merging #46 into master will increase coverage by +1.27% as of b6d5a00

@@            master     #46   diff @@
======================================
  Files           14      14       
  Stmts          756     788    +32
  Branches       152     158     +6
  Methods          0       0       
======================================
+ Hit            591     626    +35
+ Partial         40      38     -2
+ Missed         125     124     -1

Review entire Coverage Diff as of b6d5a00

Powered by Codecov. Updated on successful CI builds.

[lopuhin]

Hm, strange that this https://github.com/TeamHG-Memex/undercrawler/pull/46/files#diff-cd4de073722fd256d3d944b28a9c88baR95 is not covered, may be something fishy here, I was sure it must be covered...

[kmike]

is the docstring still valid?

[lopuhin]

Nice catch, thanks! Fixed, I think now the only restriction is a single authorization domain per spider, and now it should be easier to relax if needed.

[lopuhin]

I think it's ready now @kmike !

I tried to come up with a scrapy API that would simplify this case, but did not come up with anything good. One small thing that could simplify it is making crawler.engine.crawl(request, spider) more obvious, perhaps as spider.crawl(request), but this does not change much. A more powerful thing would be to add a DelayRequest exception, so instead of adding request to self._queue it would be queued somewhere in scrapy, but then we should also have some way to say when to re-process this (original) requests, and it also must be composable, so that several middlewares can use it, so it starts looking more complicated than explicitly queueing requests in the middleware.

[kmike]

Can we solve it without dropping the request and maintaining a queue ourselves? Downloader middleware support returning Deferreds from process_request; see e.g. https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/robotstxt.py implementation.

[lopuhin]

It looks much nicer, I'll try, thanks!

[lopuhin]

It tool a while to get used to such style, but at the end it's better, I think. The only gotcha is that at one point the traceback was incorrect, is it worth a bug report?

[kmike]

Yeah, bad tracebacks worth a bug report.

[lopuhin]

Yep, filed scrapy/scrapy#1948