Starting a container via the systemd socket for podman causes all socket calls to hang after that (podman 4.6.0 and systemd 252-16)

[carlosrodfern]

systemd version the issue has been seen with

252

Used distribution

CentOS Stream 9

Linux kernel version used

uname -a Linux localhost.localdomain 5.14.0-333.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jun 28 09:47:27 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

Systemd keeps handling podman socket requests

Unexpected behaviour you saw

Systemd appears to see like if podman.service is still on receiving socket requests but in reality is podman is exited.

Steps to reproduce the problem

The environment is a CentOS Stream 9 VM (e.g. qcow2). On a non-root user.

1. Install specifically this version (the version 4.6.0-3 has other issue https://bugzilla.redhat.com/show_bug.cgi?id=2231975): sudo dnf install podman-2:4.6.0-1.el9
2. Ensure systemd 252-15 is the one installed: sudo dnf install systemd-252-15.el9
3. Enable socket systemctl --user enable --now podman.socket
4. Start nginx podman --url unix://run/user/$(id -u)/podman/podman.sock run --name nginx-test -p 8080:80 -d docker.io/nginx
5. List podman --url unix://run/user/$(id -u)/podman/podman.sock ps. Run it multiple times. They all work.
6. Remove test container: podman container rm -f nginx-test
7. Update systemd to 252-16: sudo dnf install systemd-252-16.el9
8. Reboot
9. Start nginx podman --url unix://run/user/$(id -u)/podman/podman.sock run --name nginx-test -p 8080:80 -d docker.io/nginx
10. List podman --url unix://run/user/$(id -u)/podman/podman.sock ps. This one hangs.

Additional program output to the terminal or log subsystem illustrating the issue

What I'm seeing is that the processes started in step 9 are included in the `podman.service` cgroup, and even though podman itself already exited, the `podman.service` appears as "active (running)", which seems to prevent systemd from starting podman to attend the subsequent socket requests.


podman.service - Podman API Service
     Loaded: loaded (/usr/lib/systemd/user/podman.service; disabled; preset: disabled)
     Active: active (running) since Mon 2023-08-14 18:03:31 EDT; 9min ago
TriggeredBy: ● podman.socket
       Docs: man:podman-system-service(1)
    Process: 1516 ExecStart=/usr/bin/podman $LOGGING system service (code=exited, status=0/SUCCESS)
   Main PID: 1516 (code=exited, status=0/SUCCESS)
      Tasks: 13 (limit: 10774)
     Memory: 30.2M
        CPU: 75ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/podman.service
             ├─1526 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -r 3 -e 4 --netns-type=path /run/user/1000/n>
             ├─1529 rootlessport
             └─1538 rootlessport-child

[dtardon]

This has been fixed by #28000.