tmpfiles: Support cleaning up only top-level directories, based on all timestamps within them

[joshtriplett]

Component

systemd-tmpfiles

Is your feature request related to a problem? Please describe

Cleaning up temporary files is useful. However, cleaning up individual temporary files on a file-by-file basis can be error-prone and hard to debug, by removing individual files within a directory tree based on the timestamps of that particular file.

Describe the solution you'd like

I would like to have an option/modifier character within the "Age" field of a tmpfiles.d entry to look at the timestamps of every file and directory under each top-level directory, and clean up the entire directory as a unit if and only if all files in it are older than the specified age.

For instance, with this modifier applied to /tmp and an age of 10d, tmpfiles would look at each top-level directory in /tmp (e.g. /tmp/abc and /tmp/xyz) as a unit. If a single file anywhere in /tmp/abc is newer than 10d, tmpfiles will not touch anything in /tmp/abc. If every file and directory in /tmp/xyz is older than 10d, tmpfiles will remove the entire /tmp/xyz directory recursively.

This would treat directories as a whole as something that has value as a whole, rather than something that has value with individual files removed. It's much easier to debug "my entire temporary chroot dir got removed because it was old and unused" than it is to debug "a few scattered files got removed because those files hadn't been touched recently".

Describe alternatives you've considered

No response

The systemd version you checked that didn't have the feature you are asking for

256

[barak]

This is a great idea. I'd consider expanding the request to making automatic deletion a bit more delicate in a number of respects.

• The above "treat directories as a whole" is one such.
• Another would be using statx so files which are old but have an open file descriptor by some live process (what you'd see in fuser FILE) are never eligible for automatic deletion.
• Another case: it would not be very nice if you start a big job which uses lots of temporary files and runs for a couple days, you pause the computer in the middle because you're leaving on vacation, you come back after six relaxing weeks on the beach, resume your system ... and a bunch of the big job's files are immediately deleted due to age. The obvious fix there is that the age should not be wall clock, but rather time the computer was actually running: time suspended, or arguably just turned off, should not count.

[bluca]

• Another would be using statx so files which are old but have an open file descriptor by some live process (what you'd see in fuser FILE) are never eligible for automatic deletion.

There is already a more appropriate protocol via flock, as described in the manpage

• Another case: it would not be very nice if you start a big job which uses lots of temporary files and runs for a couple days, you pause the computer in the middle because you're leaving on vacation, you come back after six relaxing weeks on the beach, resume your system ... and a bunch of the big job's files are immediately deleted due to age. The obvious fix there is that the age should not be wall clock, but rather time the computer was actually running: time suspended, or arguably just turned off, should not count.

That would be basically unpredictable and confusing. Just don't use temporary directories for things that you cannot afford to lose and recreate, that's what they are for

[barak]

That would be basically unpredictable and confusing.

Wait; you're saying "suspend laptop while on vacation; big job keeping working files in /var/tmp/ gets them deleted when unsuspended" is the predictable and non-confusing option? Because as someone who uses their laptop for actual scientific data processing, I would find that extremely confusing.

I'm also lost on your statx() vs flock comment. What the flock thing is for is saying "please do not delete this file while the process holding the flock is alive". Okay. But deleting a file that is currently open by a process, even if that process has not done the flock thing, seems rather perverse. It is hard for me to imagine circumstances in which a user would say "oh good, the system deleted /var/tmp/bigjob-output while bigjob was running and had that file open."

[bluca]

That would be basically unpredictable and confusing.

Wait; you're saying "suspend laptop while on vacation; big job keeping working files in /var/tmp/ gets them deleted when unsuspended" is the predictable and non-confusing option? Because as someone who uses their laptop for actual scientific data processing, I would find that extremely confusing.

Yes, mostly because it's not real, it's just a made-up strawman for the sake of argumenting. If that was a real case, one can simply change the configuration to suit the particular situation, it's not really rocket science, it's a config file.

I'm also lost on your statx() vs flock comment. What the flock thing is for is saying "please do not delete this file while the process holding the flock is alive". Okay. But deleting a file that is currently open by a process, even if that process has not done the flock thing, seems rather perverse. It is hard for me to imagine circumstances in which a user would say "oh good, the system deleted /var/tmp/bigjob-output while bigjob was running and had that file open."

That's yet another imaginary situation. How do we know? Because it ignores that atime/ctime/mtime are all considered, so it's not real.

Once again, please stop making things up, and provide real situations and real bugs. The titular issue shows a real use case and has merits. If it's not worth your time to actually try the things you are referencing, for real, then it's not worth our time to consider them.

[barak]

It is a very common idiom in scientific computing to use /var/tmp/ for stuff that you don't want backed up, but also don't want deleted. For example, if you download a large dataset, people will typically put it in /var/tmp/. It is seriously routine; ask in some scientific computing forum if you'd like to confirm this. I understand you believe it's a bad habit. Maybe so. But where else would you suggest? Often /home/ is backed up, and the only other standard user-writable locations are /tmp and /var/tmp. I'll start creating /var/scratch/ on machines I manage for this purpose, but that's not standard.

In deference to this established user base, I'd suggest some care be taken. Maybe creating a file /var/tmp/foo/.preserve should shield /var/tmp/foo from reaping? This would be a minimal change.

[bluca]

Please spend at least some time to understand what you are talking about, you are proposing things that already exist, and have existed for more than 10 years. Just look at the manpage for the x type entry.

[joshtriplett]

@barak @bluca Can we please not derail this issue with different feature requests? I can appreciate the additional features you're asking for, @barak, and I do think both of them are potentially reasonable and practical feature requests, but they aren't this feature request. Please file a separate feature request, so that this request doesn't become an amalgam of several different things, and so that each feature request can be evaluated separately and someone can write an implementation for it separately.

[barak]

@barak @bluca ... Please file a separate feature request

Good point. Will do.