Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document that the crawl order is BFO for small numbers of start requests #3621

Merged
merged 1 commit into from Jul 8, 2019

Conversation

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Feb 12, 2019

Fixes #1739

@codecov
Copy link

@codecov codecov bot commented Feb 12, 2019

Codecov Report

Merging #3621 into master will decrease coverage by 0.55%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3621      +/-   ##
==========================================
- Coverage   85.59%   85.03%   -0.56%     
==========================================
  Files         164      164              
  Lines        9545     9545              
  Branches     1430     1430              
==========================================
- Hits         8170     8117      -53     
- Misses       1128     1175      +47     
- Partials      247      253       +6
Impacted Files Coverage Δ
scrapy/core/downloader/handlers/s3.py 62.9% <0%> (-32.26%) ⬇️
scrapy/utils/boto.py 46.66% <0%> (-26.67%) ⬇️
scrapy/core/downloader/tls.py 77.5% <0%> (-12.5%) ⬇️
scrapy/extensions/feedexport.py 79.81% <0%> (-5.17%) ⬇️
scrapy/core/downloader/handlers/http11.py 89.92% <0%> (-2.62%) ⬇️
scrapy/core/scraper.py 86.48% <0%> (-2.03%) ⬇️
scrapy/pipelines/files.py 65.38% <0%> (-1.16%) ⬇️

docs/faq.rst Show resolved Hide resolved
@Gallaecio Gallaecio force-pushed the bfo-for-few-requests branch from 16272b6 to 0411dda Feb 22, 2019
ejulio
ejulio approved these changes Feb 22, 2019
Copy link
Contributor

@ejulio ejulio left a comment

Looks good to me 😄

docs/faq.rst Outdated
in most cases. If you do want to crawl in true `BFO order`_, you can do it by
in most cases.

However, when the number of start request is small it always `crawls in BFO
Copy link
Member

@kmike kmike Jun 25, 2019

I don't think it is correct: if there is a single request in start_requests, but then spider schedules many requests (e.g. after extracting links), behavior would be as usual, maybe besides first few requests.

@Gallaecio Gallaecio force-pushed the bfo-for-few-requests branch from 0411dda to 783de8a Jul 8, 2019
@Gallaecio
Copy link
Member Author

@Gallaecio Gallaecio commented Jul 8, 2019

I’ve switched to a completely new wording that I hope is more accurate and clear.

@kmike kmike merged commit a97a9b5 into scrapy:master Jul 8, 2019
3 checks passed
@kmike
Copy link
Member

@kmike kmike commented Jul 8, 2019

Thanks @Gallaecio, I like the clarity of the explanation!

@kmike kmike added this to the v1.7 milestone Jul 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants