Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document CloseSpider extension better #2759

Closed
redapple opened this issue May 26, 2017 · 7 comments
Closed

Document CloseSpider extension better #2759

redapple opened this issue May 26, 2017 · 7 comments

Comments

@redapple
Copy link
Contributor

@redapple redapple commented May 26, 2017

See #2748 (comment)

@kmike 's details:

When CLOSERSPIDER_ITEMCOUNT limit is reached, Scrapy stops the spider gracefully; it means requests which are currently in the downloader queue (up to CONCURRENT_REQUEST requests) are still processed. This is the expected behavior.

@redapple
Copy link
Contributor Author

@redapple redapple commented Jul 27, 2017

Closed via #2777

@redapple redapple closed this Jul 27, 2017
@OfirD1
Copy link
Contributor

@OfirD1 OfirD1 commented Oct 4, 2020

See #2748 (comment)

@kmike 's details:

When CLOSERSPIDER_ITEMCOUNT limit is reached, Scrapy stops the spider gracefully; it means requests which are currently in the downloader queue (up to CONCURRENT_REQUEST requests) are still processed. This is the expected behavior.

I think it's worth adding the same documentation for CLOSESPIDER_PAGECOUNT and CLOSESPIDER_ERRORCOUNT. If it were there, I wouldn't have to test it myself to verify :)

@eLRuLL
Copy link
Member

@eLRuLL eLRuLL commented Oct 4, 2020

@OfirD1 yeah I think we should put this part:

Requests which are currently in the downloader queue (up to :setting: CONCURRENT_REQUESTS requests) are still processed.

in a more generic part, as all closing behaviors still continue processing pending requests.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Oct 5, 2020

@OfirD1 Could you please open a new ticket about it?

@OfirD1
Copy link
Contributor

@OfirD1 OfirD1 commented Oct 6, 2020

@Gallaecio, you mean open a new issue? or just edit it here and hit "propose changes"? I'm not sure what "open a new ticket" is, but I want to learn!

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Oct 6, 2020

Sorry, I meant a new issue.

You can of course create a pull request the way you describe if you are willing to handle the change yourself. But if you do it that way, make sure you replace 2.3 with master in the URL.

@OfirD1
Copy link
Contributor

@OfirD1 OfirD1 commented Oct 6, 2020

Done.

#4836

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.