Add systemd service and timer for running 'podman image prune -a'

[johanneskastl]

Reason

I am running rootless podman and have the podman-auto-update service and timer active. I found that while the podman-auto-update service is containing a podman image prune command, it was not able to properly clean up the outdated images filling up the disk.

Running image prune with -a solved this without causing harm.

I think having a system (or user) service for running this command would come in handy.

Side note:
Not sure if there should be a service for running podman image prune without -a for people not using the auto-update service.

Does this PR introduce a user-facing change?

This PR adds a systemd (system and user) service and timer for running podman image prune -a.

add systemd service and timer to run 'podman image prune -a'

[giuseppe]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe, johanneskastl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [giuseppe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Luap99]

I don't think this is a good idea to install by default.

At the very least this will totally race against the auto-update service. There is no image pin/locking so if auto-update pulls a new one restarts the container unit the image will have no users and will be pruned which is totally incorrect.

And even without auto-update this will be racy for any podman run $IMAGE where $IMAGE is not here locally and is just being pulled. There are ton of windows where prune can delete stuff that is in fact just about to be used. As I don't think running this on a timer by default is good. I have seen distros such as debian enabling all our units so this can have bad consequences.

IMO all users must be fully aware of when pruning would happen so I rather have them do this explicitly themselves.

[Luap99]

This should not have a install section, only the timer should be enabled

[johanneskastl]

Thanks for the review. I just copied the existing podman-auto-update.service.in service which also has an Install section.
Removed in ed68145

[giuseppe]

I don't think this is a good idea to install by default.

I can imagine some users find it useful (I've ended up with something similar for my own use). My LGTM was about adding it, but not installing it by default which as you say can have some unexpected consequences.

[johanneskastl]

At the very least this will totally race against the auto-update service. There is no image pin/locking so if auto-update pulls a new one restarts the container unit the image will have no users and will be pruned which is totally incorrect.

Good point. I can added a comment in the service file stating this problematic points.

AFAIK there is no systemd mechanism to prevent one unit running while another one is running.

Is there any special reason why -a is not enabled in the podman-auto-update service's ExecStartPost? If not then I would open a new PR adding the flag in that service.

And even without auto-update this will be racy for any podman run $IMAGE where $IMAGE is not here locally and is just being pulled. There are ton of windows where prune can delete stuff that is in fact just about to be used. As I don't think running this on a timer by default is good. I have seen distros such as debian enabling all our units so this can have bad consequences.

Supplying a service does not mean that all services should be enabled. If a distribution chooses to do so, it is up to their packagers to explain why. But not providing a service because one distribution does funny things makes it harder for all users on other distributions that do not enable services/timers by default (e.g. openSUSE, Fedora, etc.)

IMO all users must be fully aware of when pruning would happen so I rather have them do this explicitly themselves.

True, but this service would only make it easier for users to do that. I is not enabled by default and needs user action to be enabled.

I just think it is a waste of time to have each user have to write their own service.

Kind Regards,
Johannes

[Luap99]

Good point. I can added a comment in the service file stating this problematic points.

AFAIK there is no systemd mechanism to prevent one unit running while another one is running.

I mean there is After=/Before= but I am not sure how this plays together with the timers.

IMO a comment in the service files solves nothing, like what normal user would look into a service files for such comment. Instead some online "guides" will tell users to turn on the service and then we get to deal with all the hard to debug bug reports where random images are deleted.
And if the out of the box shipped service files are conflicting than this is really a rather bad user experience.

Is there any special reason why -a is not enabled in the podman-auto-update service's ExecStartPost? If not then I would open a new PR adding the flag in that service.

Because it will delete all unused user images! For an auto update that seem a rather unexpected side effect. It is really the same issue I have with this unit. Some parallel podman command could fail because of it and it the worse case we delete critical user data without their consent.

Supplying a service does not mean that all services should be enabled. If a distribution chooses to do so, it is up to their packagers to explain why. But not providing a service because one distribution does funny things makes it harder for all users on other distributions that do not enable services/timers by default (e.g. openSUSE, Fedora, etc.)

Sure I don't disagree on that part. I have also pushed debian to no enable all our services by default due other bugs so maybe that is no longer a concern.

[johanneskastl]

IMO a comment in the service files solves nothing, like what normal user would look into a service files for such comment. Instead some online "guides" will tell users to turn on the service and then we get to deal with all the hard to debug bug reports where random images are deleted. And if the out of the box shipped service files are conflicting than this is really a rather bad user experience.

I get your point about user errors.

Is there any special reason why -a is not enabled in the podman-auto-update service's ExecStartPost? If not then I would open a new PR adding the flag in that service.

Because it will delete all unused user images!

I need to read up on the fine details of podman image prune, but I thought that was the whole point of having the prune command included in the auto-update service? Delete images that are no longer in use, i.e. ones that were used before the auto-update?

In my case the auto-update service updated the images, but did not remove anything and my disk started filling up...

I am not sure what the difference between a dangling image and an unused one is, but in my case they weren't dangling and hence not being removed...

For an auto update that seem a rather unexpected side effect. It is really the same issue I have with this unit. Some parallel podman command could fail because of it and it the worse case we delete critical user data without their consent.

Yes, but that is something the user needs to be aware of. And as this timer/service needs to be explicitly enabled by a user it is their responsibility?

Of course the name and description can be adjusted to make it clear that ALL UNUSED IMAGES WILL BE DELETED or similar. But in the end it is the user's decision, isn't it?
(Of course, keeping aside the "distribution XYZ enables things by default" discussion).

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[Luap99]

In my case the auto-update service updated the images, but did not remove anything and my disk started filling up...

I am not sure what the difference between a dangling image and an unused one is, but in my case they weren't dangling and hence not being removed...

dangling is a image without a name, i.e. where podman images shows <none> as repository and tag.
podman images --filter dangling=true should show exactly what is considered dangling.
Generally for podman auto-update I assume it pull the exact same image name again and that then causes the new image to get the name while the old image got its name removed and thus should end up dangling.
If that doesn't happen then something seems very odd about podman auto-update.

Of course the name and description can be adjusted to make it clear that ALL UNUSED IMAGES WILL BE DELETED or similar. But in the end it is the user's decision, isn't it?

I am not sure where such description would be visible to regular users. I mean I am not strictly against such service per se but if the out of the box experience will cause conflicts/race conditions against the auto update service then I think this is a problem and not worth including.
Now one thing I haven't tried is what if you add order dependencies such as After=podman-auto-update.service? Would that be enough to guarantee the prune service is not running while the auto-update is? In such case I would have much less problems with that.

ps: this change currently doesn't do anything either since oyu must add the new file in the Makefile for it to be installed, see install.systemd target

[johanneskastl]

I am not sure where such description would be visible to regular users. I mean I am not strictly against such service per se

I think that is the main point, if the user does not read what is in the service description...

but if the out of the box experience will cause conflicts/race conditions against the auto update service then I think this is a problem and not worth including. Now one thing I haven't tried is what if you add order dependencies such as After=podman-auto-update.service? Would that be enough to guarantee the prune service is not running while the auto-update is?

IMHO and take this with a grain of salt: After and Before are only used for ordering during startup and shutdown. And only influence the start time. So just because the first unit is started before the second, does not mean that the first is finished already.

In such case I would have much less problems with that.

Having a Conflicts would be nice to prevent any funny cornercases, yes.

ps: this change currently doesn't do anything either since oyu must add the new file in the Makefile for it to be installed, see install.systemd target

Thanks for the hint, I added what I hope might be enough (and I hope vim did not break the whitespace...)

[Luap99]

but if the out of the box experience will cause conflicts/race conditions against the auto update service then I think this is a problem and not worth including. Now one thing I haven't tried is what if you add order dependencies such as After=podman-auto-update.service? Would that be enough to guarantee the prune service is not running while the auto-update is?

IMHO and take this with a grain of salt: After and Before are only used for ordering during startup and shutdown. And only influence the start time. So just because the first unit is started before the second, does not mean that the first is finished already.

AFAIK that is not the case for type=oneshot units, https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html

Behavior of oneshot is similar to simple; however, the service manager will consider the unit up after the main process exits. It will then start follow-up units.

I have not tested that but we rely on that for quadlet as well I think.