Checkout directory growing huge

[Hritik14]

During initial import/improve, my /tmp/xyz/checkout dir is taking over 20G. It would be nice to be a little gentle and/or be able to set an limit.

Further, after import/improve process ends (with errors, currently), the checkout directory persists - would it make sense to delete the directory before exit ?

[keshav-space]

Further, after import/improve process ends (with errors, currently), the checkout directory persists - would it make sense to delete the directory before exit ?

@Hritik14 we have added support for on_failure, which runs if the importer or improver fails. All pipelines that download data has to implement on_failure to ensure that the downloaded data is removed even if the pipeline fails.

vulnerablecode/vulnerabilities/pipelines/__init__.py

Lines 152 to 161 in 30b3ef6

	def on_failure(self):
	"""
	Tasks to run in the event that pipeline execution fails.
	Implement cleanup or other tasks that need to be performed
	on pipeline failure, such as:
	- Removing cloned repositories.
	- Deleting downloaded archives.
	"""
	pass

@ziadhany there is a bug in alpine importer pipeline we should run clean_downloads once collect_and_store_advisories has complete.

vulnerablecode/vulnerabilities/pipelines/v2_importers/alpine_linux_importer.py

Lines 47 to 48 in 30b3ef6

	cls.clone,
	cls.collect_and_store_advisories,

[ziadhany]

@keshav-space Okay, I will double-check that all imports have clean_downloads func and fix any that don't.

[Hritik14]

There could be cases where clean_downloads never gets a chance to fire for reasons not in our control (see for eg: #2283). Clearing temporary files in such case would be very helpful.