After upgrading from 4.4.1 -> 4.6.1, cannot run rootless containers

[paulcalabro]

Issue Description

After upgrading Podman from v4.4.1 to v4.6.1, I am no longer able to run rootless containers.

Steps to reproduce the issue

Steps to reproduce the issue

1. Upgrade from 4.4.1 to 4.6.1 (I've also tried later releases)
2. Execute: podman run alpine:3.19.1 (I've also tried other images)

Describe the results you received

This is the error message:

Error: crun: mount cgroup2 to sys/fs/cgroup: Permission denied: OCI permission denied

Describe the results you expected

I expected the container to run successfully like it had done in previous releases.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 606c693de21bcbab87e31002e46663c5f2dc8a9b'
  cpuUtilization:
    idlePercent: 99.01
    systemPercent: 0.36
    userPercent: 0.64
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "9.3"
  eventLogger: journald
  freeLocks: 2047
  hostname: hostXYZ
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 19000
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 12345
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 5.14.0-362.18.1.el9_3.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 3754311680
  memTotal: 8013557760
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-2.el9_3.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.7-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /run/user/12345/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/12345/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 2144571392
  swapTotal: 2147479552
  uptime: 51h 36m 59.00s (Approximately 2.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/userXYZ/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/userXYZ/test
  graphRootAllocated: 4284481536
  graphRootUsed: 3947290624
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/12345/containers
  transientStore: false
  volumePath: /home/userXYZ/test/volumes
version:
  APIVersion: 4.6.1
  Built: 1705652564
  BuiltTime: Fri Jan 19 03:22:44 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

SELinux is in permissive mode atm:

$ getenforce
Permissive

User storage configuration:

[storage]
driver = "overlay"
rootless_storage_path = "/home/userXYZ/test"

User container configuration:

[containers]
netns = "private"
userns = "auto"

Additional information

• Does not happen on a fresh RHEL 9.3 install.
• Seems like another user might be experiencing a similar issue as well.
• I'm seeing the following system call fail: (let me know if you need more)

57027<3> 09:12:06 openat2(14</home/userXYZ/test/overlay/7aafa869370091d457f1d08a67094f3dfb651bc4de832cc9f7f75ee28edda9e0/merged>, "sys/fs/cgroup", {flags=O_RDONLY|O_CLOEXEC|O_PATH, resolve=RESOLVE_IN_ROOT}, 24) = 6</home/userXYZ/test/overlay/7aafa869370091d457f1d08a67094f3dfb651bc4de832cc9f7f75ee28edda9e0/merged/sys/fs/cgroup>

57027<3> 09:12:06 mount("cgroup2", "/proc/self/fd/6", "cgroup2", MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REC|MS_RELATIME, NULL) = -1 EACCES (Permission denied)

...

57025<3> 09:12:06 ioctl(2<pipe:[14706360]>, TCGETS, 0x7ffd39fdca30) = -1 ENOTTY (Inappropriate ioctl for device)
57025<3> 09:12:06 write(2<pipe:[14706360]>, "mount `cgroup2` to `sys/fs/cgroup`: Permission denied\n", 54) = 54

...

57024<conmon> 09:12:06 <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 57025
57024<conmon> 09:12:06 read(10<pipe:[14706360]>, "mount `cgroup2` to `sys/fs/cgroup`: Permission denied\n", 8191) = 54
57024<conmon> 09:12:06 write(2</dev/null>, "[conmon:w]: runtime stderr: mount `cgroup2` to `sys/fs/cgroup`: Permission denied\n\n", 83) = 83

...

57017<podman> 09:12:06 <... sendmsg resumed>) = 503
57017<podman> 09:12:06 write(2</dev/pts/1>, "Error: crun: mount `cgroup2` to `sys/fs/cgroup`: Permission denied: OCI permission denied\n", 90 <unfinished ...>

• Can run any container if I do either of the following:
  • Configure netns to host
  • Configure userns to host
• Can trigger the failure also by creating a new network and using that:

  > podman network create foobar
  foobar
  
  > podman run --rm --network=foobar alpine:3.19.1 date
  Error: crun: mount `cgroup2` to `sys/fs/cgroup`: Permission denied: OCI permission denied
  

[baude]

if this is rhel, why not file an official support issue? there is some preference for this over using upstream github for these kinds of things.

[antonkoenig]

@baude Well, we filed one for our issue. MSFT support tried to close the issue before it could reach Red Hat support. It's still an open issue. (Update: Same situation like @paulcalabro described. For our case, the support engineer did not identify the root cause so far.)

[paulcalabro]

@baude We did, however, the support engineer we worked with was unable to identity the root cause of the issue. We're still trying to figure it out.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

We only support the latest version upstream, so either try to reproduce with the latest upstream version and/or try to get your support ticket escalated if this is big problem for you.