rootless: permission denied when trying to write to /run/user

[tobwen]

/kind bug

Description
When running podman from an unprivileged user (uid=1001(tobwen) gid=1001(tobwen) groups=1001(tobwen)), podman tried to write to /run/user, where the user doesn't have permission on Debian.

Steps to reproduce the issue:

1. no podman-config file exists (I'm using switches only)
2. all the dependencies (crun etc.) are installed and the paths are correct

/home/tobwen/podman/usr/local/bin/podman \
--conmon /home/tobwen/podman/usr/local/bin/conmon \
--network-cmd-path /home/tobwen/podman/usr/local/bin/slirp4netns \
--runtime /home/tobwen/podman/usr/local/bin/crun \
--storage-driver overlay \
--storage-opt 'overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs'

Describe the results you received:

ERRO[0000] could not get runtime: error generating default config from memory: cannot mkdir /run/user/0/libpod: mkdir /run/user/0/libpod: permission denied

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Error: could not get runtime: error generating default config from memory: cannot mkdir /run/user/0/libpod: mkdir /run/user/0/libpod: permission denied

Output of podman info --debug:

Error: could not get runtime: error generating default config from memory: cannot mkdir /run/user/0/libpod: mkdir /run/user/0/libpod: permission denied

Package info (e.g. output of rpm -q podman or apt list podman):

built from source with a stack, which worked in the past

Additional environment details (AWS, VirtualBox, physical, etc.):
Debian 10 on Virtual Box 6 on Windows 10 (all 64-bit)

[tobwen]

Looks like a regression, but it does not solve the problem.

https://github.com/containers/libpod/blob/master/troubleshooting.md#4-rootless-could-not-get-runtime---database-configuration-mismatch

[tobwen]

I fixed it: Damn you, cgroupfs v2!

doesn't work

su tobwen

works

su -l tobwen

Should we add this to https://github.com/containers/libpod/blob/master/troubleshooting.md ?

[rhatdan]

How did cgroupv2 get in the way?

[rhatdan]

su tobwen would either not set the environment of the user or the environment would stay the same as the caller. I would figure the former. So since the environ was not set, then podman is blowing up since it is thinking that the container is running as root?

[tobwen]

How did cgroupv2 get in the way?

Before switching to cgroupv2, I didn't need to use su -l. It worked with simple su.

su tobwen would either not set the environment of the user or the environment would stay the same as the caller. I would figure the former.

No, I think the latter, since:

$ id
uid=1001(tobwen) gid=1001(tobwen) groups=1001(tobwen)

But the DBUS (I think it was DBUS) env variables are missing. Those get populated only with linger + su -l.

So since the environ was not set, then podman is blowing up since it is thinking that the container is running as root?

Exactly.

So two things are needed when not login directly to the user:

loginctl enable-linger $user
su -l $user

Shall I add it to troubleshooting.md via PR?

[mheon]

Loginctl I believe is already mentioned; if it's not, it needs to be.

[rhatdan]

Linger is covered in the troubleshooting @tobwen if you want to add more detail, please open a PR.

[tarjei]

Hi, I cannot find anything in the troubleshooting guide mentioned above for solving this problem. (Error: mkdir /run/user/1001/containers: permission denied).

I'm having the same problem on Ubuntu 18.04. What is the correct solution?

[lzap]

I also do not understand, I am seeing the same behavior on Fedora 34 Server when I am signed in as a regular user with uid/gid of 1001:

[lzap@nuc test]$ podman --version
podman version 3.1.2

[lzap@nuc test]$ id
uid=1001(lzap) gid=1001(lzap) skupiny=1001(lzap),4(adm),10(wheel),978(znc),1000(data) kontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[lzap@nuc test]$ podman-compose build
using podman version: podman version 3.1.2
podman build -t a_container -f ./docker/Dockerfile .
Error: mkdir /run/user/1000: permission denied
125

I could workaround it by creating /run/user/XXXX manually but that does not seem right.

[mheon]

Did you copy a Podman configuration from another machine? Alternatively, is it possible that the XDG_RUNTIME_DIR environment variable is set incorrectly (to another user's UID)?

[lzap]

I did reinstall of F33 and then upgrade to F34. However I carried my home directory from the old installation over, so yes, there could be some leftovers in .local or .config. And I can confirm that previously my user was 1000 but on this installation that would be 1001.

[mheon]

Unfortunately we don't provide an easy way of migrating from one UID to another - the best way is probably to rm -rf ~/.local/share/containers/ and start from scratch.

[axelsimon]

Quick and dirty workaround, for someone else landing here:
sudo mkdir /run/user/1000 && sudo chmod 700 /run/user/1000 && sudo chown $(whoami): /run/user/1000
Just creates the old user directory and gives your user the right to use said directory.

[rhatdan]

# machinectl shell -q test@.host ...

Would setup the proper shell environment also.

[ellcs]

@axelsimon wrote:

Unfortunately we don't provide an easy way of migrating from one UID to another - the best way is probably to rm -rf ~/.local/share/containers/ and start from scratch.

If you can not delete the files because of permission issues and can not use podman unshare, because you get the permission error again: Use the unshare command directly. Fork and map the user to root:

$ unshare -fr 
$ id
uid=0(root) gid=0(root) groups=0(root),65534(nobody)
$ rm -rf ~/.local/share/containers/

[KasperSkytte]

Simply running loginctl enable-linger $USER fixed this for me.