Failed to add podman to systemd sandbox cgroup

/kind bug

Description

Rootless podman throws errors and is not useable

Steps to reproduce the issue:

1. Install podman on Debian 10 or Ubuntu 20.04 as described https://podman.io/getting-started/installation

2. Create user and login with su -l username` or via ssh username@example.com

3. Run podman ps or podman run docker.io/hello-world

Describe the results you received:

$ podman ps
WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ podman run docker.io/hello-world
WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
Trying to pull docker.io/library/hello-world:latest...
  Running image docker://hello-world:latest is rejected by policy.
Error: Source image rejected: Running image docker://hello-world:latest is rejected by policy.

Describe the results you expected:
rootless podman should work as expected

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.14
Built:        Wed Dec 31 19:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.24, commit: '
  cpus: 1
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: oberon
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 4.19.0-6-amd64
  linkmode: dynamic
  memFree: 92901376
  memTotal: 1035145216
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.16.3-fd58-dirty
      commit: fd582c529489c0738e7039cbc036781d1d039014
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/podman-run-1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.4
  swapFree: 126803968
  swapTotal: 134213632
  uptime: 25h 46m 1.65s (Approximately 1.04 days)
registries: {}
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/test/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /tmp/podman-run-1002/containers
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.0-rc1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 3.0.0~0.4.rc1 amd64 [installed]
podman/unknown 3.0.0~0.4.rc1 arm64
podman/unknown 3.0.0~0.3.rc1 armhf
podman/unknown 3.0.0~0.4.rc1 ppc64el

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

KVM, Debian 10

[mheon]

That warning looks nonfatal - I think we have a separate issue with signature policy here. @rhatdan Concur?

[rhatdan]

Yup, although the the issue seems to be the XDG_RUNTIME_DIR is set incorrectly to /run/user/0 which could be causing problems. I believe this might be a case where podman is getting confused because it is not a full login session.

[rhatdan]

@idolux If you login to the session directly does Podman work? Rather then coming in via root.

No, does not work

test@oberon:~$ echo $XDG_RUNTIME_DIR
/run/user/1002
test@oberon:~$ podman version
WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.14
Built:        Wed Dec 31 19:00:00 1969
OS/Arch:      linux/amd64
test@oberon:~$ podman ps
WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
test@oberon:~$ podman run docker.io/hello-world
WARN[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied 
Trying to pull docker.io/library/hello-world:latest...
  Running image docker://hello-world:latest is rejected by policy.
Error: Source image rejected: Running image docker://hello-world:latest is rejected by policy.

$XDG_RUNTIME_DIR is empty if I switch from root to test with sudo -i -u test or su -l test

[rhatdan]

@giuseppe What do you think is going on here?

printenv | grep DBUS
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/3267/bus

For some reason this is still set to the root dbus.

printenv | grep DBUS

DBUS_SESSION_BUS_ADDRESS is empty on my machine

[rhatdan]

Ok then podman is generating this itself.

	if _, found := os.LookupEnv("DBUS_SESSION_BUS_ADDRESS"); !found {
		sessionAddr := filepath.Join(os.Getenv("XDG_RUNTIME_DIR"), "bus")
		if _, err := os.Stat(sessionAddr); err == nil {
			os.Setenv("DBUS_SESSION_BUS_ADDRESS", "unix:path="+sessionAddr)
		}
	}

I just noticed (never tested before actually) that the policy validation error happens with root too

# podman run docker.io/hello-world
Trying to pull docker.io/library/hello-world:latest...
  Running image docker://hello-world:latest is rejected by policy.
Error: Source image rejected: Running image docker://hello-world:latest is rejected by policy.

XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS is empty

[giuseppe]

Ok then podman is generating this itself.

	if _, found := os.LookupEnv("DBUS_SESSION_BUS_ADDRESS"); !found {
		sessionAddr := filepath.Join(os.Getenv("XDG_RUNTIME_DIR"), "bus")
		if _, err := os.Stat(sessionAddr); err == nil {
			os.Setenv("DBUS_SESSION_BUS_ADDRESS", "unix:path="+sessionAddr)
		}
	}

we should probably limit this logic to rootless.

EDIT: it looks like we do.

So the issue seems to be with XDG_RUNTIME_DIR pointing to the wrong location.

One thing we could do is to check the ownership for XDG_RUNTIME_DIR and refuse to run if it is not owned by the current user, so we will catch these kind of errors

[giuseppe]

proposed fix: containers/common#422