[Bug]: podman-system-generator failed with exit status 1

[barzog]

Issue Description

After upgrading to podman 4.4 (ArchLinux using latest podman 4.4.0-2) on every systemctl daemon-reload I see following in logs:
systemd[1311594]: /usr/lib/systemd/system-generators/podman-system-generator failed with exit status 1

I suspect that this caused by introduced in 4.4 quadlet integration. But my system does not have any needed dirs or files for quadlet (/etc/containers/systemd), I rely on manually created systemd files.

Steps to reproduce the issue

Just run systemctl daemon-reload or /usr/lib/podman/quadlet /etc/containers/systemd ; echo $?

Describe the results you received

systemd[1311594]: /usr/lib/systemd/system-generators/podman-system-generator failed with exit status 1

Describe the results you expected

No errors shown in log files

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon принадлежит conmon 1:2.1.5-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
  cpuUtilization:
    idlePercent: 99.5
    systemPercent: 0.22
    userPercent: 0.28
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: nas
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.19.13-arch1-1-barzog
  linkmode: dynamic
  logDriver: journald
  memFree: 631918592
  memTotal: 16490774528
  networkBackend: cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun принадлежит crun 1.8-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns принадлежит slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8584425472
  swapTotal: 8589930496
  uptime: 577h 56m 42.00s (Approximately 24.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 9
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 226424152064
  graphRootUsed: 56040407040
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 79
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.0
  Built: 1675466908
  BuiltTime: Sat Feb  4 02:28:28 2023
  GitCommit: 3443f453e28169a88848f90a7ce3137fc4a4bebf-dirty
  GoVersion: go1.20
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Arch Linux, kernel 5.19.13

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[vrothberg]

@barzog thanks for reaching out. Can you spot any traces via journalctl?

Cc: @alexlarsson @ygalblum

[barzog]

Nope, except
journalctl -n 100 -f
фев 06 16:11:27 nas sudo[1314055]: barzog : TTY=pts/0 ; PWD=/home/barzog ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload
фев 06 16:11:27 nas sudo[1314055]: pam_unix(sudo:session): session opened for user root(uid=0) by barzog(uid=1001)
фев 06 16:11:27 nas systemd[1]: Reloading.
фев 06 16:11:27 nas systemd[1314062]: /usr/lib/systemd/system-generators/podman-system-generator failed with exit status 1.
фев 06 16:11:27 nas sudo[1314055]: pam_unix(sudo:session): session closed for user root

That's all what outputted to journald when running sudo systemctl daemon-reload

[heftig]

> /usr/lib/podman/quadlet -dryrun x y z 
quadlet-generator[63616]: No files to parse from [/etc/containers/systemd /usr/share/containers/systemd]

(Exit code 1)

When there are no configuration files to parse, quadlet should exit successfully (0), not with a failure.

[vrothberg]

Thanks folks! I opened #17398 to fix the issue. We'll prepare a Podman v4.4.1 soon.