CentOS Container with systemd = "D-Bus connection ..."

[dschier-wtd]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running CentOS Containers with enabled systemd not properly working on Fedora 31 as host.

Steps to reproduce the issue:

As rootless, using CentOS image (daemon)

1. podman run -d --systemd=true --name c7-systemd centos:7 /sbin/init
2. podman exec c7-systemd yum install httpd -y
3. podman exec c7-systemd systemctl start httpd.service

Failed to get D-Bus connection: Operation not permitted
Error: non zero exit code: 1: OCI runtime error

As root, using CentOS image (daemon):

1. podman run -d --systemd=true --name c7-systemd centos:7 /sbin/init
2. podman exec c7-systemd yum install httpd -y
3. podman exec c7-systemd systemctl start httpd.service

Failed to get D-Bus connection: Operation not permitted
Error: non zero exit code: 1: OCI runtime error

As rootless, using CentOS image (interactive):

1. podman run -it --systemd=true centos:7 /sbin/init

Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

As root, using CentOS image (interactive):

1. sudo podman run -it --systemd=true centos:7 /sbin/init

Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

Describe the results you received:

Running centos with systemd is running into oci issues, as far as i can see and seems to have improper permissions somehow.

Describe the results you expected:

Running centos with systemd should be working with the --systemd=true switch

Output of podman version:

podman version 1.8.0

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 9744019456
  MemTotal: 16602099712
  OCIRuntime:
    name: crun
    package: crun-0.12.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.1
      commit: df5f2b2369b3d9f36d175e1183b26e5cee55dd0a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8371826688
  SwapTotal: 8371826688
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: nb01
  kernel: 5.4.17-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 6h 0m 7.4s (Approximately 0.25 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /var/home/dschier/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /var/home/dschier/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 13
  RunRoot: /run/user/1000
  VolumePath: /var/home/dschier/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

[mheon]

Can you run sudo podman run -it --systemd=true centos:7 /sbin/init --log-level debug (this should enable debug logging in systemd itself, which should tell us what's gone wrong)

[mheon]

Oh, and are you running as root, or without?

[dschier-wtd]

@mheon, thanks for your awesomely fast response. I have tested both.

1. root container

sudo podman run -it --systemd=true --log-level debug centos:7 /sbin/init
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata-fc:[/usr/bin/kata-fc] kata-qemu:[/usr/bin/kata-qemu] kata-runtime:[/usr/bin/kata-runtime] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is being used 
DEBU[0000] cached value indicated that native-diff is not being used 
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument 
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] Found CNI network cni-podman1 (type=bridge) at /etc/cni/net.d/cni-podman1.conflist 
INFO[0000] Found CNI network cni-podman2 (type=bridge) at /etc/cni/net.d/cni-podman2.conflist 
WARN[0000] Default CNI network name podman is unchangeable 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]docker.io/library/centos:7" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] Using bridge netmode                         
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] created container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has work directory "/var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has run directory "/var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata" 
DEBU[0000] New container created "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" 
DEBU[0000] container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" has CgroupParent "machine.slice/libpod-2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] Made network namespace at /var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2 for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
INFO[0000] About to add CNI network lo (type=loopback)  
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/4BXP7WXWETD2SSPFLIW6JRFPWE,upperdir=/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/diff,workdir=/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/work,context="system_u:object_r:container_file_t:s0:c223,c315" 
DEBU[0000] mounted container "2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf" at "/var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/merged" 
DEBU[0000] Created root filesystem for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf at /var/lib/containers/storage/overlay/98a54959b5b8ed38f4c2818c6a000473e8ad1e88f837578b5916568bf61aa56a/merged 
INFO[0000] Got pod network &{Name:sad_chandrasekhar Namespace:sad_chandrasekhar ID:2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf NetNS:/var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2 Networks:[] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:f6:58:ed:cc:0d:f0 Sandbox:} {Name:veth978775bd Mac:82:ef:62:75:63:8c Sandbox:} {Name:eth0 Mac:be:37:a7:0d:7a:cb Sandbox:/var/run/netns/cni-00944fa5-e76f-d964-4ff3-d8b3a0ad57c2}], IP:[{Version:4 Interface:0xc0005d2718 Address:{IP:10.88.0.19 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf to machine.slice:libpod:2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf at /var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf -u 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata -p /var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog -t --conmon-pidfile /var/run/containers/storage/overlay-containers/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf.scope 
DEBU[0000] Received: 19469                              
INFO[0000] Got Conmon PID as 19465                      
DEBU[0000] Created container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf in OCI runtime 
DEBU[0000] Attaching to container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] connecting to socket /var/run/libpod/socket/2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf/attach 
DEBU[0000] Received a resize event: {Width:106 Height:54} 
DEBU[0000] Starting container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf with command [/sbin/init] 
DEBU[0000] Started container 2f91ebeb351d592cf5276156bfa502b62738cd1c0b2b2ec3e4e4097cbb1debcf 
DEBU[0000] Enabling signal proxying                     
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

2. rootless container

podman run -it --systemd=true --log-level debug centos:7 /sbin/init
DEBU[0000] Reading configuration file "/var/home/dschier/.config/containers/libpod.conf" 
DEBU[0000] Merged system config "/var/home/dschier/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /var/home/dschier/.local/share/containers/storage/volumes docker://  crun map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd  /var/home/dschier/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/dschier/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/dschier/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/dschier/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/dschier/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] Reading configuration file "/var/home/dschier/.config/containers/libpod.conf" 
DEBU[0000] Merged system config "/var/home/dschier/.config/containers/libpod.conf": &{{false false false true true true} 0 {   [] [] []} /var/home/dschier/.local/share/containers/storage/volumes docker://  crun map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd  /var/home/dschier/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048 shm journald  ctrl-p,ctrl-q false false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/dschier/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/dschier/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/dschier/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/dschier/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/centos:7" 
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] parsed reference into "[overlay@/var/home/dschier/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]@5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] exporting opaque data as blob "sha256:5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd" 
DEBU[0000] created container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has work directory "/var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has run directory "/run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata" 
DEBU[0000] New container created "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" 
DEBU[0000] container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" has CgroupParent "user.slice/user-1000.slice/user@1000.service/user.slice/libpod-b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14.scope" 
DEBU[0000] Handling terminal attach                     
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-82407548-13f1-996e-df2a-7b778125c0d6 for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/dschier/.local/share/containers/storage/overlay/l/ARMEVO4WOHWQ5SOH5P5M46ZUPG,upperdir=/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/diff,workdir=/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/work,context="system_u:object_r:container_file_t:s0:c641,c875" 
DEBU[0000] mounted container "b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14" at "/var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-82407548-13f1-996e-df2a-7b778125c0d6 tap0 
DEBU[0000] Created root filesystem for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 at /var/home/dschier/.local/share/containers/storage/overlay/ece07aa152dcd4949b45c6efe7bfab46cd5c1044147a8e2bea9864208bc339cb/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 to user.slice:libpod:b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] Created OCI spec for container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 at /var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 -u b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 -r /usr/bin/crun -b /var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata -p /run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/pidfile -l k8s-file:/var/home/dschier/.local/share/containers/storage/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog -t --conmon-pidfile /run/user/1000/overlay-containers/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/dschier/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14]"
DEBU[0000] Received: 20183                              
INFO[0000] Got Conmon PID as 20179                      
DEBU[0000] Created container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 in OCI runtime 
DEBU[0000] Attaching to container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14/attach 
DEBU[0000] Received a resize event: {Width:106 Height:54} 
DEBU[0000] Starting container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 with command [/sbin/init] 
DEBU[0000] Started container b2697f383f7313bb747f894724d5c42319b57145f7a94e42ce0ebd42cba3ff14 
DEBU[0000] Enabling signal proxying                     
Failed to mount cgroup at /sys/fs/cgroup/systemd: No such file or directory
[!!!!!!] Failed to mount API filesystems, freezing.

[mheon]

Ah, sorry, I think you misunderstood - systemd also uses --log-level=debug to print debug logs, so we want it at the end of the command, so it's an argument to systemd, not Podman

[mheon]

(From looking at the logs, I don't see anything confirming we're in systemd mode, but checking the code, I don't see any indication that we actually log when we are - an oversight on our part)

[dschier-wtd]

Ahhh, understood. Unfortunately setting "--log-level=debug" for systemd in centos does not give any more details.

Maybeeee, this is os-related. I have tested Fedora and Debian in the meanwhile and both are "somewhat" working.

[mheon]

Ahh - are you on Fedora 31?

I believe there's a known issue with CentOS 7 (where systemd is too old to support cgroups v2) on top of Fedora 31 (defaults to cgroups v2)

[dschier-wtd]

that may be a thing, will test it on the cgroup-v1 machine.

[dschier-wtd]

ok, running

podman run -it --systemd=true --privileged centos:7 /sbin/init

and 

sudo podman run -it --systemd=true --privileged centos:7 /sbin/init

are working with cgroups1. Furthermore, fedora and debian images are working almost perfect, too ;)

That was very helpful, thank.

If I can add something here or support you otherwise, please feel free to give me a ping.

[rhatdan]

Yes, sadly we can not fix this issue. RHEL7/Centos7 systemd containers can not run on a cgroup V2 machine, The systemd is too old.