Podman fails with permission denied error

[call-a3]

/kind bug

Description

Steps to reproduce the issue:

1. Install podman from package source http://download.opensuse.org/repositories/home:/alvistack/xUbuntu_20.04/

2. Attempt to execute a podman command as a non-root user such as "podman version" or "podman login --get-login registry-name"

Describe the results you received:
Error: error opening "/etc/cni/net.d/cni.lock": permission denied

Describe the results you expected:
For podman to execute my command as a non-root user

Additional information you deem important (e.g. issue happens only occasionally):
Commands do succeed when executing as root (obviously...)

Output of podman version:

Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.7

Built:      Thu Jan  1 00:00:00 1970
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 4
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: masked
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.4.0-99-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 13676335104
  memTotal: 16726163456
  networkBackend: cni
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.1-0ubuntu2~20.04.1
      spec: 1.0.2-dev
      go: go1.13.8
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 527h 28m 53.95s (Approximately 21.96 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: mirror.gcr.io
    Prefix: docker.io
  docker.io/library:
    Blocked: false
    Insecure: false
    Location: docker.io/library
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: mirror.gcr.io/library
    Prefix: docker.io/library
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman -a
Listing... Done
podman/unknown,now 100:4.0.1-1 amd64 [installed]
podman/unknown 100:4.0.0-1 amd64
podman/unknown 100:3.4.4-1 amd64
podman/unknown 100:3.4.3-1 amd64
podman/unknown 100:3.4.2-1 amd64
podman/unknown 100:3.3.1-1 amd64
podman/unknown 100:3.3.0-1 amd64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[flouthoc]

Hi @call-a3, Thanks for creating the issue.

The output for podman info which you have pasted belongs to root users but your issue says your are getting the error for rootless user. Could you also share output of podman info from rootless users.

For root users your user must have permission to /etc/cni/net.d/cni.lock for podman, root to work.

Try checking permission for the path and do podman system reset after correcting it.

For me it looks like

-rw-r--r--. 1 root root   0 Jan  8 02:44 cni.lock

and everything about that as

drwxr-xr-x. 1 root root

[call-a3]

Hi @flouthoc , thanks for getting back to me.

Running podman info or podman info --debug as a rootless user also results in the output

Error: error opening "/etc/cni/net.d/cni.lock": permission denied

I've checked and the lock file is indeed owned by root

$ ls -al /etc/cni/net.d/
total 8
drwxr-xr-x 2 root root 4096 Mar  2 09:20 .
drwxr-xr-x 3 root root 4096 Mar  2 09:20 ..
-rw-r--r-- 1 root root    0 Mar  2 09:20 cni.lock

I don't think I've run any podman commands after, so it's unclear to me why this lock file is here, why it is owned by root or why rootless podman needs access to it.

I've attempted to run sudo podman system reset ( running that command as rootless gives me the same error ) but that does not seem to influence the lock file.

Should I just delete the lock file manually as root and try to run a podman command as rootless again? Or should the lock file have different permissions to allow rootless to modify it?

[Luap99]

Please check your containers.conf files for network_config_dir it should be commented out by default.
The following locations are used /usr/share/containers/containers.conf /etc/containers/containers.conf and ~/.config/containers/containers.conf

[djarbz]

I am running into the same issue on Debian 11.

From what I can tell, everyone should have read permissions on the lock file.

test@2006-ct:~$ podman info
Error: error opening "/etc/cni/net.d/cni.lock": permission denied
test@2006-ct:~$ ls -lah /etc/cni/net.d/
total 1.5K
drwxr-xr-x 2 root root 3 Mar  3 17:33 .
drwxr-xr-x 3 root root 3 Mar  3 17:33 ..
-rw-r--r-- 1 root root 0 Mar  3 17:33 cni.lock

network_config_dir is set only in the /usr/ location and is not commented out by default.

ansible@2006-ct:~$ cat /usr/share/containers/containers.conf | grep network_config_dir
network_config_dir = "/etc/cni/net.d"

If I comment this line out, then podman info seems to work.

[Luap99]

It looks like you use the debian package, please report this as bug there since it seems to ship a incorrect containers.conf. Unless there is a specific reason all fields should be commented out, network_config_dir should definitely not be set by default since it will not work for rootless and it would use the wrong directory for the netavark network backend.

[djarbz]

@call-a3 Try with the latest version that was pushed to that repo last night.
I tried on a fresh install and I am no longer receiving this error.
You may need to uninstall and delete /usr/share/containers/containers.conf and then reinstall podman.

[Luap99]

Closing since this is a configuration problem and there is nothing we can do upstream.