Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. #1282

Merged
merged 3 commits into from Jul 2, 2015

Conversation

@otherchirps
Copy link
Contributor

@otherchirps otherchirps commented Jun 5, 2015

Hi folks,

I noticed a scraper I was working with would spawn enough items to blow well past my assigned MEMUSAGE_LIMIT_MB before the MEMUSAGE extension ran its next scheduled check.

I thought it might be useful to be able to alter the memory use check frequency, if you wanted, to have a better chance of a graceful shutdown.

Kept the default as it was, at 60.0 seconds.

Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.

Scope: ``scrapy.extensions.memusage``

The frequence which the current memory usage will be checked against the

This comment has been minimized.

@kmike

kmike Jun 5, 2015
Member

  • Frequency is an inverse of interval.
  • Even though option name has 'SECONDS' in it, it'd be good to mention the number is in seconds.

This comment has been minimized.

@otherchirps

otherchirps Jun 6, 2015
Author Contributor

Ah, now I understand. And there was a typo as well, which you gracefully ignored. Thanks. :) Will push a change very soon.

@@ -36,6 +36,7 @@ def __init__(self, crawler):
self.limit = crawler.settings.getint('MEMUSAGE_LIMIT_MB')*1024*1024
self.warning = crawler.settings.getint('MEMUSAGE_WARNING_MB')*1024*1024
self.report = crawler.settings.getbool('MEMUSAGE_REPORT')
self.check_interval = crawler.settings.getfloat('MEMUSAGE_CHECK_INTERVAL_SECONDS', 60.0)

This comment has been minimized.

This comment has been minimized.

@otherchirps

otherchirps Jun 6, 2015
Author Contributor

Will do.

@kmike
Copy link
Member

@kmike kmike commented Jun 5, 2015

+1 to add this feature 👍

@kmike
Copy link
Member

@kmike kmike commented Jun 6, 2015

Looks good, thanks!

@kmike kmike changed the title Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. [MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. Jun 6, 2015
dangra added a commit that referenced this pull request Jul 2, 2015
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
@dangra dangra merged commit 3fc4e0b into scrapy:master Jul 2, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants