Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Allow core Scheduler priority queue customization #1822

Merged
merged 1 commit into from Mar 2, 2016

Conversation

@nyov
Copy link
Contributor

@nyov nyov commented Feb 27, 2016

It seems there are use-cases for replacing the PriorityQueue in the Scheduler.
This adds a setting to do so via SCHEDULER_PRIORITY_QUEUE.

@codecov-io
Copy link

@codecov-io codecov-io commented Feb 27, 2016

Current coverage is 83.18%

Merging #1822 into master will not affect coverage as of 1b239bb

Powered by Codecov. Updated on successful CI builds.

@@ -13,9 +12,11 @@

class Scheduler(object):

def __init__(self, dupefilter, jobdir=None, dqclass=None, mqclass=None, logunser=False, stats=None):
def __init__(self, dupefilter, jobdir=None, logunser=False, stats=None,
pqclass=None, dqclass=None, mqclass=None):

This comment has been minimized.

@kmike

kmike Mar 1, 2016
Member

New argument order is more clean, but changing the order of arguments is technically backwards-incompatible.

This comment has been minimized.

@nyov

nyov Mar 1, 2016
Author Contributor

Yeah, I guess if someone subclassed Scheduler and overrode from_crawler for example, that might bite.
Well, if you say it's unsafe for the keyword args, I'll add the pqclass after stats.

Since you say "technically backwards-incompatible", should I change it or not?
Probably it should be backwards-compatible if it goes into 1.1, but maybe not necessarily if it only hits 1.2 ?

Hmm, let me know.

This comment has been minimized.

@kmike

kmike Mar 1, 2016
Member

It is better to make it backwards compatible again.
There are no plans to break compatibility in 1.2 :)

This comment has been minimized.

@nyov

nyov Mar 1, 2016
Author Contributor

Would it help if I pass the class instancing args in from_crawler as kwargs; so it might be re-ordered in the distant future? Or won't that happen anyway. (as dqclass=dqclass and so on)

This comment has been minimized.

@kmike

kmike Mar 1, 2016
Member

+1 to use kwargs in from_crawler; it won't help with backwards compatibility (because users may still use *args), but it'll make code cleaner and less likely to break.

@nyov nyov force-pushed the nyov:nyov/scheduler branch from a1d1094 to 2a6524e Mar 1, 2016
@kmike kmike changed the title Allow core Scheduler priority queue customization [MRG+1] Allow core Scheduler priority queue customization Mar 2, 2016
@kmike
Copy link
Member

@kmike kmike commented Mar 2, 2016

👍

redapple added a commit that referenced this pull request Mar 2, 2016
[MRG+1] Allow core Scheduler priority queue customization
@redapple redapple merged commit 9f4fe5d into scrapy:master Mar 2, 2016
1 of 2 checks passed
1 of 2 checks passed
codecov/patch 86.00% of diff hit (target 100.00%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@nyov nyov deleted the nyov:nyov/scheduler branch Mar 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants