podman docker socket works differently when journald is set to MaxLevelStore=Warning

[waffshappen]

Issue Description

This is a follow up on a Woodpecker CI Bug Report that i am not sure where its caused but whatever causes it: its cursed in some way.

I am running a CI Container (Woodpecker) as a non-root user with access to the user podman socket (podman-docker installed). This works fine in the default setup - containers start, stop and send their output to the ci server just fine.

However when

/etc/systemd/journald.conf.d/10-journald.conf
MaxLevelStore=warning

everything stops working. Containers are never detected as "finished" and their logs, which do exist, are only sent when cancelling the process from cli or from the woodpecker gui.

For Reference:
woodpecker-ci/woodpecker#2035

Steps to reproduce the issue

Steps to reproduce the issue

1. Set up a new user and podman-docker with systemctl --user enable podman.socket
2. Start woodpecker ci with access to the user socket
3. Limit journald to MaxLeveLStore=warning

Describe the results you received

Woodpecker never gets notified of containers finishing their task

Describe the results you expected

The journald log level should have no difference in how podman works, especially when container logs are set to "json" format.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 95.3
    systemPercent: 1.34
    userPercent: 3.36
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "38"
  eventLogger: journald
  freeLocks: 2046
  hostname: tobimain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 1024288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 1024288
      size: 65536
  kernel: 6.4.4-200.fc38.x86_64
  linkmode: dynamic
  logDriver: json-file
  memFree: 2294239232
  memTotal: 67324641280
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 400h 45m 36.00s (Approximately 16.67 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/woodpecker/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/woodpecker/.local/share/containers/storage
  graphRootAllocated: 510481399808
  graphRootUsed: 103872135168
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /home/woodpecker/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.0
  Built: 1689942206
  BuiltTime: Fri Jul 21 14:23:26 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Fedora 38, baremetal

Additional information

No response

[Luap99]

eventLogger: journald

Juts a guess but if we use journald for events it will likely not have any events in the log and many external tool rely on these events to detect the container state changes.

So my guess is you need to change the event logger to file in containers.conf to make it work.

[waffshappen]

Can confirm that it works with

~/.config/containers/containers.conf 
[containers]
log_driver="json-file"

[engine]
events_logger="file"