Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up temporary archive file at the end of saveCache #791

Closed
rcowsill opened this issue May 5, 2021 · 0 comments · Fixed by #792
Closed

Clean up temporary archive file at the end of saveCache #791

rcowsill opened this issue May 5, 2021 · 0 comments · Fixed by #792
Labels
enhancement New feature or request

Comments

@rcowsill
Copy link
Contributor

rcowsill commented May 5, 2021

Describe the enhancement
When saving multiple caches, the temporary tar files created for each upload are left on the filesystem. With large caches the wasted space can be the difference between a successful run and an IOException: No space left on device error.

Consider a workflow that needs to save four separate caches containing 1GB each. The first call to saveCache creates a temporary archive, which could be up to 1GB in size (for uncompressible data). That temporary file is never deleted, so the subsequent calls have 1GB less SSD space available. By the end of the upload process there could be 4GB of wasted space that's unavailable for the rest of the workflow.

restoreCache already has a "finally" block to delete each temporary archive after its contents have been extracted. If this were implemented for saveCache the scenario above would use at most 1GB of space temporarily during upload. Once uploads were complete there would be no temporary files left on the SSD.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant