Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elaborated Request priority value in Scrapy docs #714

Merged
merged 1 commit into from May 7, 2014

Conversation

@andrewbaxter
Copy link

@andrewbaxter andrewbaxter commented May 7, 2014

I confirmed in the code: it looks like -priority is used to add the requests to a queuelib PriorityQueue

I tested with this code (n=2 mostly occurred first, followed by n=1 and then n=0):

from scrapy.spider import Spider
from random import shuffle
from scrapy.http import Request

class TestSpider(Spider):
    name = "test"
    start_urls = [
        'http://www.linkedin.com/company/scrapinghub?n=8',
        'http://www.linkedin.com/company/scrapinghub?n=9'
    ]

    def parse(self, response):
        reqs = [Request(
            'http://www.linkedin.com/company/scrapinghub?n=0',
            dont_filter=True,
            priority = -50,
            callback=self.parse2
        ) for i in range(10)] + [Request(
            'http://www.linkedin.com/company/scrapinghub?n=1',
            dont_filter=True,
            priority = 0,
            callback=self.parse2
        ) for i in range(10)] + [Request(
            'http://www.linkedin.com/company/scrapinghub?n=2',
            dont_filter=True,
            priority = 50,
            callback=self.parse2
        ) for i in range(10)]
        shuffle(reqs)
        return reqs

    def parse2(self, response):
        pass
Rendaw
@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented May 7, 2014

@andrewbaxter what is wrong with it?

@andrewbaxter
Copy link
Author

@andrewbaxter andrewbaxter commented May 7, 2014

There was nothing wrong but it didn't indicate how the priority value is applied to the request ordering algorithms. The commit modifies the documentation to hopefully make that clearer.

The test above is to confirm that the changes are accurate.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented May 7, 2014

Well +1

@chekunkov
Copy link
Contributor

@chekunkov chekunkov commented May 7, 2014

yep, +1 to merge this, it should be documented and priorities confuse some developers from time to time

pablohoffman added a commit that referenced this pull request May 7, 2014
Elaborated Request priority value in Scrapy docs
@pablohoffman pablohoffman merged commit 2bf09b8 into scrapy:master May 7, 2014
1 check passed
1 check passed
continuous-integration/travis-ci The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.