Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable memusage extension by default #2187

Closed
kmike opened this issue Aug 19, 2016 · 4 comments
Closed

enable memusage extension by default #2187

kmike opened this issue Aug 19, 2016 · 4 comments
Labels

Comments

@kmike
Copy link
Member

@kmike kmike commented Aug 19, 2016

What about setting MEMUSAGE_ENABLED to True by default? It will enable memusage/max value in stats for all spiders, and the cost is just one getrusage(RUSAGE_SELF) call per minute.

@kmike kmike added the discuss label Aug 19, 2016
@redapple
Copy link
Contributor

@redapple redapple commented Aug 19, 2016

@kmike , as long as we keep MEMUSAGE_LIMIT_MB default to 0, I like the idea.
I'm sensing this is related to #2173 , or?
By the way, MEMUSAGE_REPORT doesn't seem to do anything.

As a side note, we need to do a better job at reporting memory usage when limit is reached, it seems the extension only prints the engine status when email notificaations are on.

@kmike
Copy link
Member Author

@kmike kmike commented Sep 19, 2016

@redapple yep, it is related to #2173 - it helps with ad-hoc monitoring of the crawls by looking at the logs while the spider is still crawling.

@kmike
Copy link
Member Author

@kmike kmike commented Sep 19, 2016

Good catch about MEMUSAGE_REPORT and other memusage issues.
One more issue: #1329

@eliasdorneles
Copy link
Member

@eliasdorneles eliasdorneles commented Sep 19, 2016

I'd 👍 anything giving more visibility to what's happening with the spider with minimal noise & cost, like this. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.