Error: readlink: Permission denied: OCI permission denied

[edsantiago]

This is one of those flakes that appear at random, in random tests, making it almost impossible for my flake-logger to isolate. Symptom is:

Running: podman [options] run [...]  quay.io/libpod/alpine:latest [command]
Error: readlink: Permission denied: OCI permission denied

One interesting thing: I see this (sometimes, not always) with run -d, implying that the error happens before detach.

Podman UserNS support [It] podman --userns=container:CTR

• fedora-34 : int podman fedora-34 rootless host
  • PR Always spawn a cleanup process with exec #10405
    • 06-11 09:42

Podman pod create [It] podman pod container can override pod not sharing pid

• fedora-33 : int podman fedora-33 rootless host
  • PR rootless: fix fast join userns path #10611
    • 06-09 10:19
• fedora-34 : int podman fedora-34 rootless host
  • PR Always spawn a cleanup process with exec #10405
    • 06-10 15:02
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 11:23

Podman run [It] podman shared PID NS container share SELinux labels

• fedora-33 : int podman fedora-33 rootless host
  • PR container: ignore named hierarchies #10609
    • 06-10 10:43
• fedora-34 : int podman fedora-34 rootless host
  • PR Fix pre-checkpointing #10639
    • 06-10 11:01

Podman pod create [It] podman pod correctly sets up PIDNS

• fedora-34 : int podman fedora-34 rootless host
  • PR fix: uid/gid for volume mounted to existing dir #10905
    • 07-12 09:47
  • PR Fix network connect race with docker-compose #10654
    • 06-11 10:46
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR Add support for podman remote build -f - . #10651
    • 06-11 07:14

Podman top [It] podman pod top on pod with containers in same pid namespace

• fedora-33 : int podman fedora-33 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 08:28
• ubuntu-2104 : int podman ubuntu-2104 rootless host
  • PR add correct slirp ip to /etc/hosts #10684
    • 06-15 11:23

sys: podman selinux: shared context in (some) namespaces

• fedora-34 : sys podman fedora-34 rootless host
  • PR Makefile: remove install.cni #10803
    • 06-29 07:10

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

Podman top [It] podman pod top on pod with containers in same pid namespace

• fedora-33 : int podman fedora-33 rootless host
  • PR skip flaking auto-update test #11176
    • 08-10 06:00

Podman pod create [It] podman pod container can override pod not sharing pid

• fedora-34 : int podman fedora-34 rootless host
  • PR podman-remote build use .containerignore over .dockerignore  #10913
    • 07-14 08:58

Podman pod create [It] podman pod correctly sets up PIDNS

• fedora-34 : int podman fedora-34 rootless host
  • PR fix: uid/gid for volume mounted to existing dir #10905
    • 07-12 09:47

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

I've been keeping flake logs for a few months, and on August 17 I wrote a tool to grep those logs. The last occurrence of "readlink" on the same line as "Permission denied" was August 10 (already logged in comment above). It's possible that the error message changed, also possible that the issue magically fixed itself. I don't think there's anything I can do aside from close the issue.

[edsantiago]

Reopening: this is still happening.

Podman pod create [It] podman pod create with --userns=keep-id can add users

• fedora-34 : int podman fedora-34 rootless host
  • PR Exclude already built sources for static build #12235
    • 11-09 04:36
• fedora-35 : int podman fedora-35 rootless host
  • PR e2e tests: use HaveKey() and HaveLen() when possible #12482
    • 12-02 10:46

Podman pod create [It] podman pod create with --userns=auto

• fedora-35 : int podman fedora-35 rootless host
  • PR Fix possible rootless netns cleanup race #12469
    • 12-01 14:04
    • 12-01 14:04
    • 12-01 14:04

Podman pod create [It] podman pod create with --userns=keep-id

• ubuntu-2110 : int podman ubuntu-2110 rootless host
  • PR Cirrus: Update to Ubuntu 21.10 + Disable F33 #12256
    • 11-11 14:59

Podman pod create [It] podman pod create --userns=auto:size=%d

• fedora-34 : int podman fedora-34 rootless host
  • PR test/system: podman run with log-opt option #12221
    • 11-08 09:51

Two observations:

• There's an awful lot of userns in the list above. Much more than I think can be explained by coincidence.
• In logs where this happens, it seems to happen often, in other tests.

[vrothberg]

          podman pod create --userns=auto:size=%d
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:739
         
         [BeforeEach] Podman pod create
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:28
         [It] podman pod create --userns=auto:size=%d
           /var/tmp/go/src/github.com/containers/podman/test/e2e/pod_create_test.go:739
         Running: podman [options] pod create --userns=auto:size=500 --name testPod
         6092a09b3c751d0a768f5f7193034c2d793fbd3680fe7c42e344b8dcaf202fd4
         Running: podman [options] run --pod testPod quay.io/libpod/alpine:latest cat /proc/self/uid_map
                  0          1        500
         Running: podman [options] pod create --userns=auto:size=3000 --name testPod-1
         40ec609fac65ff52559ee8033d475074eb009f924d909ed11a42ab3b458a6bb9
         Running: podman [options] run --pod testPod-1 quay.io/libpod/alpine:latest cat /proc/self/uid_map
         Error: readlink: Permission denied: OCI permission denied

@giuseppe, could this be a crun issue?

[rhatdan]

No you are running a container within a Pod and already have set the userns size to 500, now you are attempting to set a container in the same user namesapce to use 3000 uids, which is not allowed.

What should happen here is --pod and userns option should conflict if container is sharing userns with the pod.