[MRG+1] Downloader-aware Priority Queue for Scrapy #3520

whalebot-helmsman · 2018-12-05T11:08:51Z

This is long long-awaited by community(#1802, #2474 and #3140) priority queue implementation for multi host case. Contrary to past proposals it is based on ~~Downloader signals~~ direct interaction with downloader, not on round-robin principle. Round-robin is much slower according to benchmarks. ~~It became possible after new signal was added in #3393~~.
Intention was to provide solution for this pain in shortest term as possible. So two problems left out of scope:

I didn't directly use Adding domain #3160 but take a lot of inspiration from it. Thanks @tianhuil. Adding domain #3160 replaces priorities by hostnames as a slot for scheduling. As a result priorities of requests does not affect scheduling. I tried to merge priority and hostname to single structure. This structure is consumable by old downstream priority queue. As a result priorities affect scheduling. Both ways are not perfect. Perfect solution is to refactor current Scheduler structure.
This new priority queue is not set as default priority queue, because it does not support CONCURRENCY_PER_IP option. To enable such support we should resolve hostnames on adding them to queue. This is deferred operation and code calling to Scheduler.enqueue_request should be ready for this situation. This is not the case now.

I run benchmarks using https://github.com/scrapy/scrapy-bench. In single host case(bookworm spider) new priority queue does not affect performance. In multi host case(broadworm spider) new priority queue improves it 10 times. Exact numbers available https://docs.google.com/spreadsheets/d/1f8iSzfmLuiG9O-mDgG4iDb4fGvAlnkC4sH_zr9Qlj-s/edit#gid=0, full logs are available https://gist.github.com/whalebot-helmsman/83ed45a1aa1b4221ab6be2f36226c5e3. Round robin based queue also does not affect single host case, but improves multi host one only 2-3x. That's why I remove round robin implementation.

Settings used for bookworm https://github.com/scrapy/scrapy-bench/blob/f03011d63f4149eeaad6dd7908ffa6fcbbb3304e/books/books/settings.py
Settings used for broadworm https://github.com/scrapy/scrapy-bench/blob/f03011d63f4149eeaad6dd7908ffa6fcbbb3304e/broad/broad/settings.py

List of changes:

added tests for current priority queue
create priority queue in scheduler using create_instance
added DOWNLOADER_SLOT constant to Downloader
added wrapper for queuelib.PriorityQueue to handle merged priorities
added DownloaderAwarePriorityQueue implementation
added tests for DownloaderAwarePriorityQueue
~~added request_left_downloader signal as response_downloaded sent only in case of successful download~~

codecov · 2018-12-05T11:29:31Z

Codecov Report

Merging #3520 into master will increase coverage by 0.55%.
The diff coverage is 96.32%.

@@            Coverage Diff             @@
##           master    #3520      +/-   ##
==========================================
+ Coverage   84.72%   85.28%   +0.55%     
==========================================
  Files         168      169       +1     
  Lines        9460     9582     +122     
  Branches     1407     1422      +15     
==========================================
+ Hits         8015     8172     +157     
+ Misses       1188     1160      -28     
+ Partials      257      250       -7

Impacted Files	Coverage Δ
scrapy/squeues.py	`100% <100%> (ø)`	⬆️
scrapy/settings/default_settings.py	`98.64% <100%> (ø)`	⬆️
scrapy/core/downloader/__init__.py	`89.05% <100%> (+0.08%)`	⬆️
scrapy/core/scheduler.py	`90.35% <86.2%> (+28.21%)`	⬆️
scrapy/pqueues.py	`98.97% <98.97%> (ø)`
scrapy/downloadermiddlewares/redirect.py	`96.77% <0%> (+0.05%)`	⬆️
scrapy/core/engine.py	`92.76% <0%> (+0.45%)`	⬆️
scrapy/dupefilters.py	`98.07% <0%> (+1.92%)`	⬆️
scrapy/spiders/crawl.py	`82.35% <0%> (+3.97%)`	⬆️
... and 2 more

scrapy/pqueues.py

whalebot-helmsman · 2018-12-17T10:46:01Z

It is not possible to check correct scheduling for DownloaderAwarePriorityQueue in integration test. A lot of things take place simultaneously - signals, site-responses and scheduling moments. I just remove integration test and keep unit test.

tianhuil · 2018-12-23T00:05:44Z

I really hope we can get this feature. It's an important feature and I just haven't had time to work on my PR.

kmike · 2018-12-26T22:35:39Z

I've started to look at the code; proposed (incomplete, and rather mechanical) cleanup so far: whalebot-helmsman#2

scrapy/pqueues.py

tests/test_scheduler.py

scrapy/pqueues.py

tests/test_scheduler.py

whalebot-helmsman · 2019-03-05T12:58:48Z

I removed commits related to request_left_downloader signal, but htey preserved in https://github.com/whalebot-helmsman/scrapy/tree/signal-backup branch

docs/topics/settings.rst

tests/test_scheduler.py

kmike · 2019-03-22T01:05:44Z

Hey @whalebot-helmsman!

I've left some comments about tests, but besides these comments the implementation looks good to me.

Could you please also clean up the commit history?

I think after fixing these minor issues with tests and cleaning up the commit history the PR is ready to merge 🎉

handle exception use O(N) instead of O(NlogN) here we have request as struct additional check for meptiness small performance improvement do not consume another request test number of responses mark requests back to 3 slots test case raise exceptions in case of missed meta add marks to requests and work only with your own requests only disk queue should obtain signals separate functions for slot rasd/write use signlas without variable stop crawler get signals in correct place logic test for download-aware priority queue update comment for structure ensure text type transform slot name to path use implicit structure use unicode type implicitly use real crawler add signals more slot accounting simple implementation of pop small slot accounting code no need for custom len function ability to call super in py27 add slots generic tests for downloader aware queue dummy implementation of crawler aware priority queue move common logic to base class rename class pass crawler to pqclass constructor do not copy quelib.PriorityQueue code add comment about new class remove obsolete function modify behaviour of queuelib.PriorityQueue to dodge very complex priority better way to get name remove obsolete commentary check boundaries function for priority convertion with known limits correct import path move file do not switch on by deffault as ip concurrency not supported set scheduler slot in case of empty slot use constant single place for added urls single place for constants use as default queue correct format for error text test migration from old version with on disk queue in these tests we have only two inflection points - jobdir and priority_queue_cls we do not need separate mock spider, use usual one do not rely on order of dict elements, imply order of list test round robiness of priority queue add comments and requirements for our magick function remove debug logging put queues into slot as we fabricate priorities we do not need special types anymore fabricate priority for priority queue more versatile priorities Scheduler class is not inflection point wrap correct types check for emptinees before initialization tests for new priority queue correct default type for startprios use exact values put common settings to base class test priorities for disk scheduler test dequeue for disk scheduler test length for disk scheduler setUp/tearDown methods for on disk schedulers new methods remove excessive line base class to handle scheduler creation correct method names test priorities deque test close scheduler on test end enqueue some requests test template for scheduler use downloader slot I/O implementation for RoundRobin queue round-robin implementation without I/O and slot detection wrappers for every disk queue class

Its initial content covers the workaround for scrapy#2473.

Fixes scrapy#1109

PEP8 fixes no need to close implicitly do not use pytest need to put it into class remove round-robin queue additional check for empty queue use pytest tmpdir fixture

https://changelog.travis-ci.com/deprecation-container-based-linux-build-environment-82037

…uler-tested

docs/topics/broad-crawls.rst

kmike · 2019-03-26T21:24:43Z

I think that's good to merge, but commit history is not super-clean still:

there are commits not related to this PR, such as afdb69e or 9c31480 - if I'm not mistaken, they'd be duplicated in Scrapy history if we merge it;
some of the commit names can be improved, e.g. 9093495 shouldn't be named "actually apply __slots__ suggestion", it is a refactoring

I'm ok with merging it as-is, but if that's not too hard, it'd be good to clean it up.

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>

Gallaecio

I’ve left a few comments, but they are mostly aesthetic. It looks good to me.

@kmike Do you think we should get a third opinion here?

docs/topics/broad-crawls.rst

docs/topics/settings.rst

scrapy/core/scheduler.py

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>

…lmsman/scrapy into round-robin-scheduler-tested

kmike · 2019-04-06T19:05:58Z

Thanks @whalebot-helmsman and everyone who provided feedback!

I think we don't need a third opinion @Gallaecio (asked @dangra a while ago about this).

whalebot-helmsman · 2019-04-08T08:11:49Z

Thanks for review @kmike, @lucywang000 and @Gallaecio for review

elacuesta · 2020-01-30T06:18:29Z

Hi there @whalebot-helmsman, I think the request_left_downloader signal could still be useful, do you have any plans to submit a new PR with it?

function to get unique file queues for any type of base queue

a25cf5c

kmike reviewed Dec 14, 2018

View reviewed changes

scrapy/pqueues.py Outdated Show resolved Hide resolved

kmike reviewed Dec 14, 2018

View reviewed changes

scrapy/pqueues.py Outdated Show resolved Hide resolved

kmike reviewed Dec 14, 2018

View reviewed changes

scrapy/pqueues.py Show resolved Hide resolved

kmike reviewed Dec 14, 2018

View reviewed changes

scrapy/pqueues.py Show resolved Hide resolved

whalebot-helmsman force-pushed the round-robin-scheduler-tested branch from ce4df9c to 1600450 Compare December 17, 2018 10:43

whalebot-helmsman force-pushed the round-robin-scheduler-tested branch from 1600450 to 5e6fe5e Compare December 17, 2018 13:42

lucywang000 reviewed Jan 4, 2019

View reviewed changes

kmike mentioned this pull request Jan 16, 2019

Scheduler: minimal interface, API docs #3559

Merged

kmike added this to the v1.7 milestone Jan 31, 2019

kmike mentioned this pull request Feb 9, 2019

Adding domain #3160

Closed

kmike reviewed Mar 21, 2019

View reviewed changes

docs/topics/settings.rst Outdated Show resolved Hide resolved

kmike reviewed Mar 21, 2019

View reviewed changes

docs/topics/settings.rst Outdated Show resolved Hide resolved

kmike reviewed Mar 21, 2019

View reviewed changes