mount through procfd: operation not permitted: OCI permission denied

[jianzhangbjz]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Build an image

[cloud-user@preserve-olm-env2 interview]$ cat test.go 
package main

import (
	"fmt"
)

func main() {
	testMap := make(map[string]int, 10)
	array := [10]string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j"}
	for i, v := range array {
		testMap[v] = i + 20
	}
	fmt.Println(len(testMap))
	for _, v := range testMap {
		fmt.Println(v)
	}

}
[cloud-user@preserve-olm-env2 interview]$ cat Dockerfile 
FROM golang:1.18
WORKDIR /app
COPY test.go ./
RUN go build test.go
ENTRYPOINT [./test]
[cloud-user@preserve-olm-env2 interview]$ podman build -t quay.io/olmqe/interview:v1 .
STEP 1/5: FROM golang:1.18
...

2. Run this image.

[cloud-user@preserve-olm-env2 interview]$ podman run quay.io/olmqe/interview:v1
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "sysfs" to rootfs at "/sys" caused: mount through procfd: operation not permitted: OCI permission denied

Describe the results you received:

[cloud-user@preserve-olm-env2 interview]$ podman run quay.io/olmqe/interview:v1
Error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "sysfs" to rootfs at "/sys" caused: mount through procfd: operation not permitted: OCI permission denied

Describe the results you expected:
Run this image successfully with rootless mode.

Additional information you deem important (e.g. issue happens only occasionally):

[cloud-user@preserve-olm-env2 interview]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 (Ootpa)

Output of podman version:

[cloud-user@preserve-olm-env2 interview]$ podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.7
Built:        Thu Jan 13 18:15:49 2022
OS/Arch:      linux/amd64

Output of podman info:

[cloud-user@preserve-olm-env2 interview]$ podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.7
Built:        Thu Jan 13 18:15:49 2022
OS/Arch:      linux/amd64
[cloud-user@preserve-olm-env2 interview]$ podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.32-1.module+el8.5.0+13852+150547f7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.32, commit: 4b12bce835c3f8acc006a43620dd955a6a73bae0'
  cpus: 8
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: preserve-olm-env2
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-287.el8.dt4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 776544256
  memTotal: 16600383488
  ociRuntime:
    name: runc
    package: runc-1.0.3-1.module+el8.5.0+13556+7f055e70.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.3
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 0
  swapTotal: 0
  uptime: 3977h 37m 17.37s (Approximately 165.71 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/cloud-user/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.8-1.module+el8.5.0+13754+92ec836b.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.8
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/cloud-user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 323
  runRoot: /run/user/1000/containers
  volumePath: /home/cloud-user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1642068949
  BuiltTime: Thu Jan 13 18:15:49 2022
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

[cloud-user@preserve-olm-env2 interview]$ rpm -q podman
podman-3.4.2-9.module+el8.5.0+13852+150547f7.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):
Full log:

[cloud-user@preserve-olm-env2 interview]$ podman run --privileged --log-level=debug  quay.io/olmqe/interview:v1
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called run.PersistentPreRunE(podman run --privileged --log-level=debug quay.io/olmqe/interview:v1) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/cloud-user/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/cloud-user/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/cloud-user/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/cloud-user/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/runc"            
INFO[0000] Found CNI network podman (type=bridge) at /home/cloud-user/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image quay.io/olmqe/interview:v1 (policy: missing) 
DEBU[0000] Looking up image "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Trying "quay.io/olmqe/interview:v1" ...      
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage ([overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d) 
DEBU[0000] Looking up image "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Trying "quay.io/olmqe/interview:v1" ...      
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage ([overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d) 
DEBU[0000] Looking up image "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Trying "quay.io/olmqe/interview:v1" ...      
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage ([overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d) 
DEBU[0000] Inspecting image 9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@968a674b3c5e0793deea81316a73df83b826e4f878be381d0a6d543484183ce7" 
DEBU[0000] exporting opaque data as blob "sha256:968a674b3c5e0793deea81316a73df83b826e4f878be381d0a6d543484183ce7" 
DEBU[0000] Looking up image "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Trying "quay.io/olmqe/interview:v1" ...      
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage 
DEBU[0000] Found image "quay.io/olmqe/interview:v1" as "quay.io/olmqe/interview:v1" in local containers storage ([overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d) 
DEBU[0000] Inspecting image 9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@968a674b3c5e0793deea81316a73df83b826e4f878be381d0a6d543484183ce7" 
DEBU[0000] exporting opaque data as blob "sha256:968a674b3c5e0793deea81316a73df83b826e4f878be381d0a6d543484183ce7" 
DEBU[0000] Inspecting image 9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d 
DEBU[0000] using systemd mode: false                    
DEBU[0000] Adding exposed ports                         
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" 
DEBU[0000] Allocated lock 14 for container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a 
DEBU[0000] parsed reference into "[overlay@/home/cloud-user/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] exporting opaque data as blob "sha256:9f59a2aedc8666729dd05a3e1091c8ab58122f5dc3ca64a9c4566f02e8e3fe8d" 
DEBU[0000] created container "2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a" 
DEBU[0000] container "2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a" has work directory "/home/cloud-user/.local/share/containers/storage/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata" 
DEBU[0000] container "2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a" has run directory "/run/user/1000/containers/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata" 
DEBU[0000] Not attaching to stdin                       
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-8c4ff88d-3b0f-c9df-1df5-9db9f8849e20 for container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=,lowerdir=/home/cloud-user/.local/share/containers/storage/overlay/l/EA2D4RPUGQSYDR2P53QXOFEPJM:/home/cloud-user/.local/share/containers/storage/overlay/l/PIY2CYW4ZAHQGD7I7VFBL5XDG6:/home/cloud-user/.local/share/containers/storage/overlay/l/F7H3XSMPPWFB3IQRNH4ZYKNQ4H:/home/cloud-user/.local/share/containers/storage/overlay/l/LCSO7FKGXL6V3GZIHAGZ6QLGSM:/home/cloud-user/.local/share/containers/storage/overlay/l/5PXNGJX6JJJAHO3A34NF4QDQUI:/home/cloud-user/.local/share/containers/storage/overlay/l/553XUIHJN24QYU37H6QE7KJ3Q3:/home/cloud-user/.local/share/containers/storage/overlay/l/GARHJ57JPYJISSS5MXOT6ZS4FU:/home/cloud-user/.local/share/containers/storage/overlay/l/3MTS6VGEYMNOUZGTH2AT5RVSOX:/home/cloud-user/.local/share/containers/storage/overlay/l/2FYBVOQPIZYCLQVCN4AFVCEN36,upperdir=/home/cloud-user/.local/share/containers/storage/overlay/bd838185037a00592f860700f04c4a931d640d565e835beaa49cd9151ad9c5d5/diff,workdir=/home/cloud-user/.local/share/containers/storage/overlay/bd838185037a00592f860700f04c4a931d640d565e835beaa49cd9151ad9c5d5/work,context="system_u:object_r:container_file_t:s0:c1022,c1023" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-8c4ff88d-3b0f-c9df-1df5-9db9f8849e20 tap0 
DEBU[0000] mounted container "2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a" at "/home/cloud-user/.local/share/containers/storage/overlay/bd838185037a00592f860700f04c4a931d640d565e835beaa49cd9151ad9c5d5/merged" 
DEBU[0000] Created root filesystem for container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a at /home/cloud-user/.local/share/containers/storage/overlay/bd838185037a00592f860700f04c4a931d640d565e835beaa49cd9151ad9c5d5/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Workdir "/app" resolved to host path "/home/cloud-user/.local/share/containers/storage/overlay/bd838185037a00592f860700f04c4a931d640d565e835beaa49cd9151ad9c5d5/merged/app" 
DEBU[0000] Created OCI spec for container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a at /home/cloud-user/.local/share/containers/storage/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a -u 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a -r /usr/bin/runc -b /home/cloud-user/.local/share/containers/storage/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata -p /run/user/1000/containers/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata/pidfile -n loving_burnell --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -l k8s-file:/home/cloud-user/.local/share/containers/storage/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/cloud-user/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a]"
INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/conmon: permission denied 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-8c4ff88d-3b0f-c9df-1df5-9db9f8849e20 for container 2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a 
DEBU[0000] unmounted container "2fa1cafb1c2cc6c86b17c2d343d7f5a4a6febdb50502412377ae6b8cb81b497a" 
DEBU[0000] ExitCode msg: "time=\"2022-08-15t11:01:17+08:00\" level=warning msg=\"unable to get oom kill count\" error=\"no directory specified for memory.oom_control\"\ntime=\"2022-08-15t11:01:18+08:00\" level=error msg=\"container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \\\"sysfs\\\" to rootfs at \\\"/sys\\\" caused: mount through procfd: operation not permitted\": oci permission denied" 
Error: time="2022-08-15T11:01:17+08:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control"
time="2022-08-15T11:01:18+08:00" level=error msg="container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting \"sysfs\" to rootfs at \"/sys\" caused: mount through procfd: operation not permitted": OCI permission denied

[jianzhangbjz]

/kind bug

[mheon]

Does running other images work? Does it work if you install crun and use --runtime crun in your podman run command?

[jianzhangbjz]

I updated Podman to 4.1.1, but still get the same error:

[cloud-user@preserve-olm-env2 interview]$ sudo dnf update podman
...
[cloud-user@preserve-olm-env2 interview]$ podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.17.7
Built:        Mon Jul 11 22:56:53 2022
OS/Arch:      linux/amd64
[cloud-user@preserve-olm-env2 interview]$ podman run  quay.io/olmqe/interview:v1
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "sysfs" to rootfs at "/sys" caused: mount through procfd: operation not permitted: OCI permission denied

[jianzhangbjz]

Thanks! Seems like it works when specifying the crun, but why?

[cloud-user@preserve-olm-env2 interview]$ dnf install crun
Repository google-cloud-sdk is listed more than once in the configuration
Error: This command has to be run with superuser privileges (under the root user on most systems).
[cloud-user@preserve-olm-env2 interview]$ sudo dnf install crun
Repository google-cloud-sdk is listed more than once in the configuration
Last metadata expiration check: 1:49:02 ago on Mon 15 Aug 2022 09:38:55 AM CST.
Dependencies resolved.
============================================================================================================================================================
 Package                   Architecture                Version                                                    Repository                           Size
============================================================================================================================================================
Installing:
 crun                      x86_64                      1.4.5-2.module+el8.6.0+15917+093ca6f8                      rhel8appstream                      209 k
Installing dependencies:
 yajl                      x86_64                      2.1.0-11.el8                                               appstream                            41 k

Transaction Summary
============================================================================================================================================================
Install  2 Packages

Total download size: 250 k
Installed size: 602 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64.rpm                                                                4.3 MB/s | 209 kB     00:00    
(2/2): yajl-2.1.0-11.el8.x86_64.rpm                                                                                          88 kB/s |  41 kB     00:00    
------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                       540 kB/s | 250 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                    1/1 
  Installing       : yajl-2.1.0-11.el8.x86_64                                                                                                           1/2 
  Installing       : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64                                                                                  2/2 
  Running scriptlet: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64                                                                                  2/2 
  Verifying        : yajl-2.1.0-11.el8.x86_64                                                                                                           1/2 
  Verifying        : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64                                                                                  2/2 

Installed:
  crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64                                         yajl-2.1.0-11.el8.x86_64                                        

Complete!
[cloud-user@preserve-olm-env2 interview]$ podman run  --runtime crun quay.io/olmqe/interview:v1
/bin/sh: 1: [./test]: not found

[jianzhangbjz]

I didn't find any introduction about the crun in the help info:

[cloud-user@preserve-olm-env2 interview]$ podman run --help|grep runtime
      --cpu-rt-runtime int                       Limit the CPU real-time runtime in microseconds

And, what's the default runtime? Thanks!

[mheon]

The default runtime everywhere except RHEL 8 is now crun. We retain runc for 8 for backwards compatibility - can't change defaults in a RHEL release mid-flight.

The --runtime flag is global, so you'll find it in man podman.

I advise that you file a Bugzilla against runc for this error.

[mheon]

(Oh, I suppose that RHEL 7 also defaults to runc now that I think about it. RHEL 9 and every other distro should be on crun)

[jianzhangbjz]

@mheon Thanks! I see now, report a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=2118231

[rhatdan]

Since this is not a podman bug, closing.