You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory #5306
changed the title [-]Rootless podman error: could not get runtime: open /proc/pid/ns/user: no such file or directory[/-][+]Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory[/+]on Feb 24, 2020
Oh I just noticed one interesting and probably important thing: The PID in the path (/proc/...) is always the same value: 31678. Of course that PID can not exist. That is probably why the thing is failing.
@rhatdan I'm running Void Linux. The Kernel supports user namespaces. The issue is probably not the namespace but podman figuring out some bogus PID when building the path.
% uname -srvmpio
Linux 5.4.21_1 #1 SMP PREEMPT Thu Feb 20 09:09:55 UTC 2020 x86_64 unknown unknown GNU/Linux
% cat /etc/os-release
NAME="void"
ID="void"
DISTRIB_ID="void"
PRETTY_NAME="void"
% ls -l /proc/self/ns
total 0
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 net -> 'net:[4026531992]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 user -> 'user:[4026531837]'
lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 uts -> 'uts:[4026531838]'
Okay, this is odd. After I've downgraded it yesterday, i've now upgraded it back to 1.8.0. Then it worked. Then I rebooted to make sure no processes from the old version are running. And it still works. Don't know what went wrong there.
Have this issue too after each reboot of the host system. Only a podman system migrate resolves this temporary (until next boot). Can't downgrade/upgrade the package, as this is a new void linux installation and only podman 1.8.0 packages is available.
@mheon one more reason why XDG_RUNTIME_DIR is may not been set properly is when one calls podman through a 3rd process like gitlab-runner. This is exactly what we do (using rootless podman from gitlab-runner through a custom executor) and at each call podman receives a new runtime directory (/tmp/custom-executor${RANDOMNUMBER}/run-${UID}) by default (which makes the solution leak podman pause processes for each job).
I do not know if you considered this in the design but please also note that /run/user/$UID supposed to be deleted when the user logs out from it's last session. From man pam_systemd:
$XDG_RUNTIME_DIR
Path to a user-private user-writable directory that is bound to the user login time on the machine. It is automatically created the first time a user logs in and removed on the user's final logout.
Also, it would be nice to handle cases like sudo -i -u and su -l the same way how normal logins... Probably this is just a dream as pam_systemd is not called and as such no one would handle the directory/variables but who knows :)
We'd really like for sudo and su to be handled correctly, but we really
need systemd to handle that on their end - not much we can do on the Podman
side of things.
On being removed at logout - that can be annoying for services that run
after the user logs out (or systemd services using the User directive) but
we can disable that behavior but enabling lingering.
On Wed, Nov 25, 2020, 08:26 Balazs Zachar ***@***.***> wrote:
@mheon <https://github.com/mheon> one more reason why XDG_RUNTIME_DIR is
may not been set properly is when one calls podman through a 3rd process
like gitlab-runner. This is exactly what we do (using rootless podman from
gitlab-runner through a custom executor) and at each call podman receives a
new runtime directory (/tmp/custom-executor${RANDOMNUMBER}/run-${UID}) by
default.
I do not know if you considered this in the design but please also note
that /run/user/$UID supposed to be deleted when the user logs out from it's
last session. From man pam_systemd:
$XDG_RUNTIME_DIR
Path to a user-private user-writable directory that is bound to the user
login time on the machine. It is automatically created the first time a
user logs in and removed on the user's final logout.
Also, it would be nice to handle cases like sudo -i -u and su -l the same
way how normal logins... Probably this is just a dream as pam_systemd is
not called and as such no one would handle the directory/variables but who
knows :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5306 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AOCDWAQR4ASXUKLT65LTSRUAWRANCNFSM4K2DOZDA>
.
Best you or we can do is handle this on reboot by using tmpfiles.d, if /tmp is not on a tmpfs.
Podman has no concept of a user logging in or out, and really should not grow this ability.
@rhatdan IMO that sounds good but only if podman does not rely on the XDG_RUNTIME_DIR variable. As we can see, that is what is causing most of the issues for users in case of rootless podman. It is not set for sudo -i, su -l and also problematic when podman started by systemd and or another process like gitlab-runner (which sets this environment variable to a random directory for each job).
In case podman would use the "tmpfiles.d interface" of systemd directly, then we could have a fixed path which works in every above mentioned case and the directory cleanup among reboots would be handled by systemd as you said.
Well if the content is always placed on /tmp, then I believe we can setup a tmpfiles.d to look for the directory podman anywhere under /tmp and remove content. We just need to have a path that the tmpfiles.d file would look for.
Please search conmon.pid in your /runroot/overlay-containers/... folder and delete the file.
Delete all content in the /tmp/runXXXX folder. Stop Start podman. No reboot is needed.
Please give feedback if this solution works for you.
Activity
[-]Rootless podman error: could not get runtime: open /proc/pid/ns/user: no such file or directory[/-][+]Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory[/+]faulesocke commentedon Feb 24, 2020
Oh I just noticed one interesting and probably important thing: The PID in the path (
/proc/...) is always the same value:31678. Of course that PID can not exist. That is probably why the thing is failing.rhatdan commentedon Feb 24, 2020
Could you check if you have a podman running as the user? If yes, could you kill it and then see if the problem goes away?
faulesocke commentedon Feb 24, 2020
No podman running, also tried restarting the box.
rhatdan commentedon Feb 24, 2020
What OS are you running podman on? Could this be your kernel does not support User Namespace?
ls -l /proc/self/ns
faulesocke commentedon Feb 24, 2020
@rhatdan I'm running Void Linux. The Kernel supports user namespaces. The issue is probably not the namespace but podman figuring out some bogus PID when building the path.
Could this issue be ulimits related?
giuseppe commentedon Feb 24, 2020
could be fixed by: #5237
faulesocke commentedon Feb 24, 2020
@giuseppe Nope. Installed 1.8.1-rc1 and the error is still there.
faulesocke commentedon Feb 24, 2020
I have downgraded a little bit. 1.7.0 is also affected but 1.6.5 works. So this got broken somewhere between 1.6.5 and 1.7.0.
faulesocke commentedon Feb 25, 2020
Okay, this is odd. After I've downgraded it yesterday, i've now upgraded it back to 1.8.0. Then it worked. Then I rebooted to make sure no processes from the old version are running. And it still works. Don't know what went wrong there.
giuseppe commentedon Feb 26, 2020
thanks for the update. Let's close the issue for now.
If it still happens, feel free to re-open it with more details
tunip commentedon Feb 26, 2020
Have this issue too after each reboot of the host system. Only a
podman system migrateresolves this temporary (until next boot). Can't downgrade/upgrade the package, as this is a new void linux installation and only podman 1.8.0 packages is available.rhatdan commentedon Feb 27, 2020
Is the runtime directory /run not on a tmpfs? We rely on the /run directory being cleaned on a reboot.
58 remaining items
Cajga commentedon Nov 25, 2020
@mheon one more reason why XDG_RUNTIME_DIR is may not been set properly is when one calls podman through a 3rd process like gitlab-runner. This is exactly what we do (using rootless podman from gitlab-runner through a custom executor) and at each call podman receives a new runtime directory (/tmp/custom-executor${RANDOMNUMBER}/run-${UID}) by default (which makes the solution leak podman pause processes for each job).
I do not know if you considered this in the design but please also note that /run/user/$UID supposed to be deleted when the user logs out from it's last session. From man pam_systemd:
Also, it would be nice to handle cases like sudo -i -u and su -l the same way how normal logins... Probably this is just a dream as pam_systemd is not called and as such no one would handle the directory/variables but who knows :)
mheon commentedon Nov 25, 2020
rhatdan commentedon Nov 25, 2020
Best you or we can do is handle this on reboot by using tmpfiles.d, if /tmp is not on a tmpfs.
Podman has no concept of a user logging in or out, and really should not grow this ability.
Cajga commentedon Nov 25, 2020
@rhatdan IMO that sounds good but only if podman does not rely on the XDG_RUNTIME_DIR variable. As we can see, that is what is causing most of the issues for users in case of rootless podman. It is not set for sudo -i, su -l and also problematic when podman started by systemd and or another process like gitlab-runner (which sets this environment variable to a random directory for each job).
In case podman would use the "tmpfiles.d interface" of systemd directly, then we could have a fixed path which works in every above mentioned case and the directory cleanup among reboots would be handled by systemd as you said.
rhatdan commentedon Nov 25, 2020
Well if the content is always placed on /tmp, then I believe we can setup a tmpfiles.d to look for the directory podman anywhere under /tmp and remove content. We just need to have a path that the tmpfiles.d file would look for.
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
contrib/build-push-ceph-container-imgs.sh: undo kubic repo hack
MICOC commentedon Aug 10, 2021
Please search conmon.pid in your /runroot/overlay-containers/... folder and delete the file.
Delete all content in the /tmp/runXXXX folder. Stop Start podman. No reboot is needed.
Please give feedback if this solution works for you.
dmchandrasinghe commentedon Jan 20, 2023
removing /tmp/runXXXX folder fixed my issue. thanks