create --init-ctr once: once again, seems to be incorrectly running

[edsantiago]

Another init-ctr flake; it actually looks identical to #11682 except for the warning, which is new:

podman make sure once container is removed
 /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:100
 
 [BeforeEach] Podman init containers
   /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:22
 [It] podman make sure once container is removed
   /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_initcontainers_test.go:100
$ podman [options] create --init-ctr once --pod new:foobar quay.io/libpod/alpine:latest bin/sh -c echo AjWwhTHctcuAxhxK > /dev/shm/XVlBzgbaiCMR
SHA1
$ podman [options] create --pod foobar -t quay.io/libpod/alpine:latest top
SHA2
$ podman [options] pod start foobar
time="2022-09-29T01:38:01Z" level=error msg="Removing container SHA1 from database: no container with ID SHA1 found in DB: no such container"
SHA3
$ podman [options] container exists SHA1
$ podman [options] pod stop foobar
SHA3
$ podman [options] pod start foobar
SHA3
$ podman [options] exec -it SHA2 cat /dev/shm/XVlBzgbaiCMR
AjWwhTHctcuAxhxK  <<<<------ UNEXPECTED! This causes test to fail, because this file should not exist

Only two instances in the past month; before that, May.

Podman init containers [It] podman make sure once container is removed

• fedora-36 : int podman fedora-36 root container
  • PR Ensure that the DF endpoint updated volume refcount #15753
    • 09-12 18:21
• ubuntu-2204 : int podman ubuntu-2204 rootless host
  • PR DO NOT MERGE: buildah vendor treadmill #13808
    • 09-28 21:46

[rhatdan]

Could this be a race condition where containers within the first Pod start have no finished closing, before the second Pod starts and ends up continuing to use the same /;dev/shm?
Is this possible? Does the infra container wait for all of the containers to actually stop or does it mark the pod stopped as soon as it sends the stop signal to its containers.

Do we need a podman pod wait?

[rhatdan]

@mheon WDYT?

[mheon]

podman pod stop guarantees that all containers in the pod have stopped, but it only cleans them up if it's explicitly asked to (and we don't ask - the false in https://github.com/containers/podman/blob/main/pkg/domain/infra/abi/pods.go#L198) - so we are racing on the post-stop cleanup process to actually unmount the SHM.

Flip the false to a true and this ought to resolve itself. #16057 should fix.

[edsantiago]

Thank you!