Add ContainerStateRemoving

[mheon]

When Libpod removes a container, there is the possibility that removal will not fully succeed. The most notable problems are storage issues, where the container cannot be removed from c/storage.

When this occurs, we were faced with a choice. We can keep the container in the state, appearing in podman ps and available for other API operations, but likely unable to do any of them as it's been partially removed. Or we can remove it very early and clean up after it's already gone. We have, until now, used the second approach.

The problem that arises is intermittent problems removing storage. We end up removing a container, failing to remove its storage, and ending up with a container permanently stuck in c/storage that we can't remove with the normal Podman CLI, can't use the name of, and generally can't interact with. A notable cause is when Podman is hit by a SIGKILL midway through removal, which can consistently cause podman rm to fail to remove storage.

We now add a new state for containers that are in the process of being removed, ContainerStateRemoving. We set this at the beginning of the removal process. It notifies Podman that the container cannot be used anymore, but preserves it in the DB until it is fully removed. This will allow Remove to be run on these containers again, which should successfully remove storage if it fails.

Fixes #3906

[mheon]

WIP: Needs tests, needs integration with ps, and there's probably locking issues I need to solve around a dependency race (container A is being removed, container B is being added with A as a dependency, A deallocates lock, B grabs lock, B is added to state, A tries to remove but fails as a dependency has been added; subsequent attempts to remove A or B will cause deadlocks as both have the same lock ID)

[rhatdan]

LGTM

[mheon]

Fixed the race by moving lock removal to after the container is removed from the database. Could potentially result in leaking locks is we get a SIGKILL precisely between the database removal and attempting to free the lock, but does solve issues where we could try to double-free a lock if RemoveContainer is run twice on the same container, once partially successful, the second time running to the end.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mheon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mheon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mheon]

Good news: this seems to work.

Bad news: it potentially introduces 4 new database writes during removal to ensure atomicity of removal operations. It doesn't seem to have a significant effect on removal time on my laptop, but four disk writes can't be free.

[mheon]

@vrothberg @baude @TomSweeneyRedHat @giuseppe PTAL

[vrothberg]

Code LGTM but tests are missing.

[TomSweeneyRedHat]

LGTM too, added tests are always welcomed.

[baude]

/lgtm