enable memusage extension by default

[kmike]

What about setting MEMUSAGE_ENABLED to True by default? It will enable memusage/max value in stats for all spiders, and the cost is just one getrusage(RUSAGE_SELF) call per minute.

[redapple]

@kmike , as long as we keep MEMUSAGE_LIMIT_MB default to 0, I like the idea.
I'm sensing this is related to #2173 , or?
By the way, MEMUSAGE_REPORT doesn't seem to do anything.

As a side note, we need to do a better job at reporting memory usage when limit is reached, it seems the extension only prints the engine status when email notificaations are on.

[kmike]

@redapple yep, it is related to #2173 - it helps with ad-hoc monitoring of the crawls by looking at the logs while the spider is still crawling.

[kmike]

Good catch about MEMUSAGE_REPORT and other memusage issues.
One more issue: #1329

[eliasdorneles]

I'd 👍 anything giving more visibility to what's happening with the spider with minimal noise & cost, like this. :)