journald log driver (silently) not working

[vrothberg]

Issue Description

In certain environments (CI example, limactl or podman-in-podman) the journald log-driver does not work. However, if journald is used by default or set by the user, Podman happily uses the driver although it doesn't work.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman run --privileged -it --rm podman
2. podman run -d --log-driver=journald alpine echo 123
3. podman logs -l

Describe the results you received

No output

Describe the results you expected

123 (or an error since the journald driver doesn't work)

podman info output

quay.io/podman/stable@sha256:72f9249a0d6db514d354d2691201c3b32bd0f49a874fd593f7420642b0cc5f80 (i.e., podman 5.5.2).

---
host:
  arch: arm64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 99.82
    systemPercent: 0.08
    userPercent: 0.1
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "42"
  eventLogger: file
  freeLocks: 2048
  hostname: 505d9a7d0038
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.13-200.fc41.aarch64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9134342144
  memTotal: 11631964160
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1.fc42.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1.fc42.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.aarch64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 12h 37m 55.00s (Approximately 0.50 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /var/lib/shared
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-3.fc42.aarch64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 98899800064
  graphRootUsed: 13399797760
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Tue Jun 24 00:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/arm64
  Version: 5.5.2

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

In the example, I forcefully set it journald while the image is configured to use k8s-file instead. However, I am seeing the very same issue in CI environments and limactl.

In limactl (fedora), I see the following log

DEBU[0000] systemd-logind: Unknown object '/'.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

https://github.com/containers/common/blob/7692941ae9c812641d33e7a9824cba3cc06a0622/pkg/config/systemd.go#L64

I don't see what else podman can do realistically to check for journald, the only thing left is directly testing by writing in podman but that spams the journal a lot which isn't good either.

And based on the code in your scenarios in all cases the default podman check shoul;d already default to file and nor journald which means your container images hard code journald via the config file so this seems to be a packing problems mostly, i.e. fedora seem to hard code journald for some reason: grep log_driver /usr/share/containers/containers.conf.

Now for reporting an error if it cannot log that I agree with but as you know log writing is done by conmon not podman so it would need to report the error back to podman which to make this a conmon problem not podman

[vrothberg]

And based on the code in your scenarios in all cases the default podman check shoul;d already default to file and nor journald which means your container images hard code journald via the config file so this seems to be a packing problems mostly, i.e. fedora seem to hard code journald for some reason: grep log_driver /usr/share/containers/containers.conf.

Good catch, that may cause the issue in limactl. Could be a nice fix for Fedora.

Now for reporting an error if it cannot log that I agree with but as you know log writing is done by conmon not podman so it would need to report the error back to podman which to make this a conmon problem not podman

Being fully ignorant toward implementations details: Podman (nor conmon) left any breadcrumb in the logs for a human to understand that the specified log-driver turns into a NOP.

I don't see what else podman can do realistically to check for journald, the only thing left is directly testing by writing in podman but that spams the journal a lot which isn't good either.

I don't have a good proposal either but I caught myself trying the test with Docker where it just worked. In fact (just tried again), the journald driver works with Docker but not with Podman (root and rootless) when running inside the Fedora lima VM. Looks like there's more to the issue than just error reporting.

[Luap99]

In fact (just tried again), the journald driver works with Docker but not with Podman (root and rootless) when running inside the Fedora lima VM. Looks like there's more to the issue than just error reporting.

I mean there are two parts, writing the logs, reading the logs. writing is done by conmon and reading by podman. So you can check if journalctl --user can see the logs. If it doesn't then there is not much podman can do. And that just reminded me of #26709 where there are seemingly cases where the logging just doesn't work properly in same cases so there could be very well a variable that we are missing here.

[vrothberg]

journalctl does see the logs

Aug 26 14:17:50 lima-fedora jolly_lalande[4571]: 123

[ninja-quokka]

I saw the same issue inside a Lima VM but forgot to report it. I assumed it was something to do with the Fedora cloud image. I could not get my non root user to be able to read or write to the journal. Running journalctl --user printed an error about permissions but even after adding my user to wheel and the specific journal groups it still wouldn't work.

I thought there was an issue there about podman not reporting it can't actually write to the log but it seemed the root issue was something related to the Fedora cloud image being used.

[ninja-quokka]

Okay after some prompting from @baude I took another look at this and reproduced it.

TLDR: Podman should return an error when it can't access the journal just like journalctl --user does.

When the user doesn't have the appropriate permissions to access the journal, Podman commands that read from the journal fail silently like podman events and podman logs:

[user@lima-fedora]~% journalctl --user
Hint: You are currently not seeing messages from the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.

[user@lima-fedora]~% podman run -d --log-driver=journald fedora echo 123
977b5462cb413abcbf9ad1532171b9a65aafcf826ad8700c4ef13446a7dfd714

[user@lima-fedora]~% podman logs -l
<no output>

[user@lima-fedora]~% podman events --since 10m
<no output>                                                                                                                                           

To resolve the issue we can add the user to the systemd-journal group and reboot:

[user@lima-fedora]~%  sudo usermod -a -G systemd-journal user
[user@lima-fedora]~%  sudo reboot

[user@lima-fedora]~% groups
user systemd-journal

[user@lima-fedora]~% journalctl --user | tail -n 2
Aug 28 12:42:31 lima-tmp-fedora systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Aug 28 12:42:31 lima-tmp-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

[user@lima-fedora]~% podman run -d --log-driver=journald fedora echo 123
8b3b2265b0128eaf46f2d5d8f44b99756175b8d2e76b0a554a30736908b5a68d

[user@lima-fedora]~% podman logs -l
123

[user@lima-fedora]~% podman events --since 10m
2025-08-28 12:34:47.400483611 +1000 AEST system refresh
8< ------------- 8< ------------- 8<

Note the issue isn't writing to the journal, it's only with reading from the journal. When we try reading from the journal but don't have permissions, rather than failing silently we should return an error like journalctl --user does.

[vrothberg]

Thanks for looking into it, @ninja-quokka 🙏

[Luap99]

Adding it to the group is likely not the proper fix as it allows the user to read all messages from the system which is likely not wanted security wise.

Did you check the journald Storage mode which is used? https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html#Storage=
I assume it would be set to auto and if /var/log/journal doesn't exist it will default to "volatile"

Note that per-user journal files are not supported unless persistent storage is enabled, thus making journalctl --user unavailable.

A quick check on the journalctl code shows this https://github.com/systemd/systemd/blob/1be088c2995f6f22767a4a58e7b51c3642930805/src/journal/journalctl-util.c#L67

And unless I am missing something these symbols are not exported so there is no way for us to see errors when reading the journald files. All we see is an empty result which is equal to not logs which is totally valid.

[ninja-quokka]

Ah! That volatile limitation is really interesting. I think you are right and I will confirm tomorrow, thank you @Luap99 !

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[DerRockWolf]

It seem that I'm also having this issue. Although journald has Storage=persistent (/var/log/journal/ has contents) and journalctl --user doesn't fail for me.

Logs can be viewed with journalctl but not with podman logs.

When running a container with e.g., --log-driver k8s-file podman logs works as expected.

podman version:

Client:       Podman Engine
Version:      5.7.0
API Version:  5.7.0
Go Version:   go1.25.4
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64

I'm on NixOS and part of the wheel group, but not part of the systemd-journal group.

Edit:
adding myself to the systemd-journal group didn't fix it. The files in var/log/journal/ID/ are all owned by root and the systemd-journal group...