podman.service: Unit process X (conmon) remains running after unit stopped.

[Aight25]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Setup a nested podman rootful container with privileged flag from a rootless user

2. Fix images

3. Start (5) containers (airflow in this case) via podman-remote from a client not running in this nested podman host.

4. A couple of days later, there are, 13055 conmon processes, controlled by the systemd service podman, which create problems with max open files etc, and makes the podmanhost container not to work that good. While doing mount | grep it complains on some lib, podman ps gives errors and so on = not strange.

Describe the results you received:

From journalctl in the nested podmanhost there are multiple logs of the following:

podman.service: Unit process 12989 (conmon) remains running after unit stopped.

podman.service: Found left-over process 17067 (conmon) in control group while starting unit. Ignoring.

This usually indicates unclean termination of a previous run, or service implementation deficiencies.

Describe the results you expected:

That the conmon process is stopped when the unit is stopped. My suspicion is that this is some sort of an api bug

Additional information you deem important (e.g. issue happens only occasionally):'
Second time I have seen it now when the customer starts using the the solution now a bit more than just random tests.

Output of podman version:

podman version 3.4.2
podman remote version 3.4.4

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 4
  distribution:
    distribution: fedora
    variant: container
    version: "34"
  eventLogger: file
  hostname: HIDDEN
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-348.2.1.el8_5.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2296111104
  memTotal: 16601096192
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4198858752
  swapTotal: 4294963200
  uptime: 158h 0m 22.97s (Approximately 6.58 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636849008
  BuiltTime: Sun Nov 14 01:16:48 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.2-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No (no stable for fc34 seems to have come out yet, but I guess it's on it's way soon, v3.4.4?

Additional environment details (AWS, VirtualBox, physical, etc.):
VM is running in vSphere with host OS, redhat-release-8.5-0.8.el8

[Aight25]

podman-remote v3.4.4-1 is not compatible with podman v3.4.2-1. While creating a new container via podman-remote, a conmon process is created, which then starts to multiply after a couple of minutes. If podman is upgraded to v3.4.4-1, then the conmon process remains just one, and is killed if the container is stopped.

Now to the question and reason to have the issue opened:
Should later podman-remote versions be compatible with previous versions?

[flouthoc]

@Aight25 Often their are changes and bug fixes where change radius affects for both client and server so its not
guaranteed that all bug fixes will be back-ported for older clients.

I think its a good idea to keep client in parity with server if you looking for bug fixes.

But I'll also tag @containers/podman-maintainers for their opinion on this.

[rhatdan]

Yes you should attempt to keep them in sync. The issue right now is we are moving very fast and they might be having difficulty in older support.

[Aight25]

What about a warning when you setup a connection with podman-remote to a podman with lower version that it isn't compatible or similar?

Thanks in advance!

[rhatdan]

@jwhonce WDYT?

[rhatdan]

@Aight25 Interested in opening a PR?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[flouthoc]

@Aight25 Producing a warning SGTM but one doubt would be when should we warn, do we want to warn on every command invocation ? Cause it might become annoying for users after a while also making an API call to check version on every command sounds expensive to me.

Ideally podman-remote info shows version for both remote and server shouldn't that be enough for checking compatibility ? I'd suggest adding a note in troubleshooting docs for this.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc What is the state of this one?

[flouthoc]

I think this needs a note on troubleshooting docs. I'll add it.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc did you ever add the note?

[flouthoc]

Added a PR for this.