_cleared_contents folder does not get deleted on clean up

[innovate-invent]

599fd97 added functionality to preserve the working directory when a job is rescheduled.

This folder is not deleted during job clean up, slowly filling the filesystem.
A side effect is when a jobs criteria is to retry indefinitely, millions of copies of the working directory are made until the server crashes due to a full filesystem.

[pcm32]

I have also recently noticed this in our 19.05 setup. Is there any update on this? Is it safe to delete _cleared_contents? Thanks!

[fgiacomoni]

Hi, any improvement from this thread. I also post on this issue in August (here) with no answer from anybody... On our WorkFlow4Metabolomics instance, this temp folder can be more than 2To...

[mvdbeek]

You should use tmpwatch to manage the job_working_directory, independently of this issue. See for instance this: https://training.galaxyproject.org/training-material/topics/admin/tutorials/maintenance/slides.html#4
See https://github.com/galaxyproject/usegalaxy-playbook/blob/master/env/main/host_vars/galaxy-web-05.tacc.utexas.edu.yml#L14 for an implementation.