Add logging functionality to memusage extension

[GeorgeA92]

Summary

To add logging functionality to memusage extension.

Motivation

Scrapy jobs with MEMUSAGE_ENABLED : True and defined MEMUSAGE_LIMIT_MB (all jobs on scrapy cloud) can be stopped early due to overuse of RAM memory and receive memusage_exceeded outcome.

First thing required to debug RAM memory leaks - is to identify.. pattern of RAM memory usage.
Is RAM usage continuously increased at higher rates during runtime?
or Is RAM usage rapidly increased over limit in last several minutes after hours or even days of stable runtime performance?
Each reason require different approaches to debug RAM memory leaks.

It will be much easier to debug this if value of self.get_virtual_size() will be added to log in _check_limit method of memusage extension

scrapy/scrapy/extensions/memusage.py

Lines 77 to 89 in 6ded3cf

	def _check_limit(self):
	if self.get_virtual_size() > self.limit:
	self.crawler.stats.set_value('memusage/limit_reached', 1)
	mem = self.limit / 1024 / 1024
	logger.error("Memory usage exceeded %(memusage)dM. Shutting down Scrapy...",
	{'memusage': mem}, extra={'crawler': self.crawler})
	if self.notify_mails:
	subj = (
	f"{self.crawler.settings['BOT_NAME']} terminated: "
	f"memory usage exceeded {mem}M at {socket.gethostname()}"
	)
	self._send_report(self.notify_mails, subj)
	self.crawler.stats.set_value('memusage/limit_notified', 1)

Describe alternatives you've considered

Applying MEMUSAGE_WARNING_MB setting to ~80-90% of MEMUSAGE_LIMIT_MB - current implementation of memusage extension warns only 1 time so it is not enough data for this.

Manually subclass memusage extension with similar changes - as well as any other option it will require to reschedule job. It may be not siutable for jobs with several days(and more) total runtime. So from this side it is preferable that it will be applied in scrapy itself and with enabled this loggin by default.

Additional context

Similar functionality previously requested here #2173

[JazzGlobal]

I have a patch ready to address this but don't have permission to push my branch. Am I doing something wrong?

[elacuesta]

@JazzGlobal I suppose you're trying to push to the main Scrapy repository, you should instead fork the repo and create a pull request from that fork.

[JazzGlobal]

@JazzGlobal I suppose you're trying to push to the main Scrapy repository, you should instead fork the repo and create a pull request from that fork.

Ah, I sure was. Thank you for the prompt and helpful response!