Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Extend the topic about broad crawls with information about memory usage #3866

Merged
merged 2 commits into from Jul 11, 2019

Conversation

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Jul 11, 2019

This is built on top of the changes proposed at #1264.

I refactored the original documentation proposed by removing the new section and, instead, extending the existing section about concurrency with memory information, and creating two separate sections about crawling in BFO order and debugging memory leaks.

I also increased reliance on existing documentation through :ref: links. For example, the section about crawling in BFO order does not elaborate on DFO vs BFO or FIFO vs LIFO, and instead expects users to follow the link to the corresponding FAQ entry for more information.

Added section on how to treat memory consumption problems of broad crawls.
@codecov
Copy link

@codecov codecov bot commented Jul 11, 2019

Codecov Report

Merging #3866 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3866   +/-   ##
=======================================
  Coverage   85.56%   85.56%           
=======================================
  Files         164      164           
  Lines        9551     9551           
  Branches     1431     1431           
=======================================
  Hits         8172     8172           
  Misses       1132     1132           
  Partials      247      247

@Gallaecio Gallaecio force-pushed the broad-crawl-memory-docs branch from 57a2e40 to b18696c Jul 11, 2019
docs/topics/broad-crawls.rst Outdated Show resolved Hide resolved
@kmike kmike changed the title Extend the topic about broad crawls with information about memory usage [MRG+1] Extend the topic about broad crawls with information about memory usage Jul 11, 2019
kmike
kmike approved these changes Jul 11, 2019
Copy link
Member

@kmike kmike left a comment

Looks good, thanks @Gallaecio and @rmuellerb! @Gallaecio feel free to merge.

@Gallaecio Gallaecio force-pushed the broad-crawl-memory-docs branch from b18696c to ed487b7 Jul 11, 2019
@Gallaecio Gallaecio merged commit c4bcfb1 into scrapy:master Jul 11, 2019
0 of 2 checks passed
@kmike kmike added this to the v1.7 milestone Jul 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants