podman remote doesn't delete cidfile

[Romain-Geissler-1A]

Issue Description

podman remote doesn't delete the cidfile (which lies on the client filesystem) when the container exits, while non remote podman does. Also, kind of linked, podman rm will always attempt to delete the cidfile on the server side, while it was initially created on the client side.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a container with podman running as a server:

rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> rm -f /tmp/podman-server.cid && podman run --detach --rm --privileged --pull=always --cidfile=/tmp/podman-server.cid -v /run/bug-resources quay.io/podman/upstream podman system service -t 0 unix:///run/bug-resources/podman.sock 
Trying to pull quay.io/podman/upstream:latest...
Getting image source signatures
Copying blob sha256:478cf87435b0e8869a055a10488139ce03218124ff4910b27da5b378446efa74
Copying blob sha256:71539fc003c4baf6506f4518c33a95ab44f981a83d1204d36228aa73037f7794
Copying blob sha256:ad5077952f523ff30dc5365f41cf2644ba63f9cf3fa5234f338d495522dfe35e
Copying blob sha256:deb8f01378c1e98c712719b4967c04f775656b153c81fb564202c48730a940bc
Copying blob sha256:3f54f76baa30373f489c91b296c10ca6cd704c800e45718eee302ee05c9e7bb4
Copying blob sha256:d8b457dfa3e42f693649243720d9ee717c76c6b54a529ecf7b6b5924820f83bd
Copying blob sha256:8972c9d0ade14f282171e36dfe3ffb00d1b256e5f872858d6ee343dbbe11bbe6
Copying blob sha256:d0e059397b9d659c78d378adbb387d08a9bb2e551e1860de449ab577c3d2f105
Copying config sha256:843c1b85879751cca5dd7cc9ec08c85f5626c26885d0bc91a0b93b6fdd824916
Writing manifest to image destination
Storing signatures
37df3e02e489d983cd0e97199c8b66880f0ec7c1bb3b984dea26b3f3cd35e96f

2. Now run another container which will use podman as a client, connecting to the above server. We see that cidfile is not removed in the end:

rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> podman run -t -i --rm --pull=always --volumes-from="$(cat /tmp/podman-server.cid)" quay.io/podman/upstream sh -c 'podman --url unix:///run/bug-resources/podman.sock run --cidfile=/tmp/cidfile --rm fedora true && ls -la /tmp/cidfile' 
Trying to pull quay.io/podman/upstream:latest...
Getting image source signatures
Copying blob sha256:478cf87435b0e8869a055a10488139ce03218124ff4910b27da5b378446efa74
Copying blob sha256:deb8f01378c1e98c712719b4967c04f775656b153c81fb564202c48730a940bc
Copying blob sha256:71539fc003c4baf6506f4518c33a95ab44f981a83d1204d36228aa73037f7794
Copying blob sha256:d8b457dfa3e42f693649243720d9ee717c76c6b54a529ecf7b6b5924820f83bd
Copying blob sha256:ad5077952f523ff30dc5365f41cf2644ba63f9cf3fa5234f338d495522dfe35e
Copying blob sha256:3f54f76baa30373f489c91b296c10ca6cd704c800e45718eee302ee05c9e7bb4
Copying blob sha256:8972c9d0ade14f282171e36dfe3ffb00d1b256e5f872858d6ee343dbbe11bbe6
Copying blob sha256:d0e059397b9d659c78d378adbb387d08a9bb2e551e1860de449ab577c3d2f105
Copying config sha256:843c1b85879751cca5dd7cc9ec08c85f5626c26885d0bc91a0b93b6fdd824916
Writing manifest to image destination
Storing signatures
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob sha256:ad5077952f523ff30dc5365f41cf2644ba63f9cf3fa5234f338d495522dfe35e
Copying config sha256:919a420d29c6f5ae0bdc8d1872387d3a878d7f69debce9e24f3f2e0506b2ba0d
Writing manifest to image destination
-rw-r--r-- 1 root root 64 Jul 28 20:55 /tmp/cidfile

3. Comparing to a container where podman is not using the client/server scheme, the cifdile is correctly removed in the end:

rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> podman run -t -i --rm --pull=always --privileged quay.io/podman/upstream sh -c 'podman run --cidfile=/tmp/cidfile --rm fedora true && ls -la /tmp/cidfile' 
Trying to pull quay.io/podman/upstream:latest...
Getting image source signatures
Copying blob sha256:478cf87435b0e8869a055a10488139ce03218124ff4910b27da5b378446efa74
Copying blob sha256:ad5077952f523ff30dc5365f41cf2644ba63f9cf3fa5234f338d495522dfe35e
Copying blob sha256:71539fc003c4baf6506f4518c33a95ab44f981a83d1204d36228aa73037f7794
Copying blob sha256:deb8f01378c1e98c712719b4967c04f775656b153c81fb564202c48730a940bc
Copying blob sha256:3f54f76baa30373f489c91b296c10ca6cd704c800e45718eee302ee05c9e7bb4
Copying blob sha256:d8b457dfa3e42f693649243720d9ee717c76c6b54a529ecf7b6b5924820f83bd
Copying blob sha256:8972c9d0ade14f282171e36dfe3ffb00d1b256e5f872858d6ee343dbbe11bbe6
Copying blob sha256:d0e059397b9d659c78d378adbb387d08a9bb2e551e1860de449ab577c3d2f105
Copying config sha256:843c1b85879751cca5dd7cc9ec08c85f5626c26885d0bc91a0b93b6fdd824916
Writing manifest to image destination
Storing signatures
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ad5077952f52 done   |
Copying config 919a420d29 done   |
Writing manifest to image destination
ls: cannot access '/tmp/cidfile': No such file or directory

4. Also, proof that podman rm when run as a client tries to delete the cidfile in the wrong machine: it tries to delete it on server side while it should be deleted on client side:

rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> podman exec "$(cat /tmp/podman-server.cid)" touch /tmp/cidfile
rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> podman run -t -i --rm --pull=always --volumes-from="$(cat /tmp/podman-server.cid)" quay.io/podman/upstream sh -c 'podman --url unix:///run/bug-resources/podman.sock run --cidfile=/tmp/cidfile fedora true && podman --url unix:///run/bug-resources/podman.sock rm "$(cat /tmp/cidfile)" && ls /tmp/cidfile' 
cat: /tmp/cidfile: No such file or directory
Trying to pull quay.io/podman/upstream:latest...
Getting image source signatures
Copying blob sha256:deb8f01378c1e98c712719b4967c04f775656b153c81fb564202c48730a940bc
Copying blob sha256:478cf87435b0e8869a055a10488139ce03218124ff4910b27da5b378446efa74
Copying blob sha256:71539fc003c4baf6506f4518c33a95ab44f981a83d1204d36228aa73037f7794
Copying blob sha256:ad5077952f523ff30dc5365f41cf2644ba63f9cf3fa5234f338d495522dfe35e
Copying blob sha256:d8b457dfa3e42f693649243720d9ee717c76c6b54a529ecf7b6b5924820f83bd
Copying blob sha256:3f54f76baa30373f489c91b296c10ca6cd704c800e45718eee302ee05c9e7bb4
Copying blob sha256:8972c9d0ade14f282171e36dfe3ffb00d1b256e5f872858d6ee343dbbe11bbe6
Copying blob sha256:d0e059397b9d659c78d378adbb387d08a9bb2e551e1860de449ab577c3d2f105
Copying config sha256:843c1b85879751cca5dd7cc9ec08c85f5626c26885d0bc91a0b93b6fdd824916
Writing manifest to image destination
Storing signatures
d2039d3c55f7c3b9a49d109d6e1a195f44ede345d3036eff7f3322ad635c164c
/tmp/cidfile
rgeissler@ncerndobedev6097:~/wk/podman (v4.4.1-rhel-patched-by-Amadeus *% u=)> podman exec "$(cat /tmp/podman-server.cid)" ls /tmp/cidfile
ls: cannot access '/tmp/cidfile': No such file or directory

I guess that if really you want to fix this, then podman remote will have to fork/exec into the background and not exit until the container actually stops. From past discussion we already had, I actually doubt you want to do this for real, but this is how docker works.

Describe the results you received

The cidfile is removed on the server side (effectively leaking it on client side).

Describe the results you expected

The cidfile should be removed on client side.

podman info output

Not needed as tested from the upstream podman image.

Podman in a container

Yes

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response