Skip to content

Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory #5306

Closed

Description

@faulesocke

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running any podman command (for example podman info or podman version) as non-root only gives me the following output:

% podman info
Error: could not get runtime: open /proc/31678/ns/user: no such file or directory

It works when running as root.

Steps to reproduce the issue:

  1. Run podman info

Describe the results you received:

Error: could not get runtime: open /proc/31678/ns/user: no such file or directory

Describe the results you expected:
The actual command output...

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

% sudo podman version
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.7
OS/Arch:            linux/amd64

Output of podman info --debug:

UNOBTAINABLE

Package info (e.g. output of rpm -q podman or apt list podman):

% xbps-query podman
architecture: x86_64
filename-sha256: 514f725a15bc57ec717606103eaedd77f603d62c3b29007166ef9d235b308ac2
filename-size: 18MB
homepage: https://podman.io/
install-date: 2020-02-24 09:11 CET
installed_size: 54MB
license: Apache-2.0
maintainer: Cameron Nemo <camerontnorman@gmail.com>
metafile-sha256: 347e1471b2966a2350df5661216db7a1f60103544ebbe406d8e4a48dde169782
pkgver: podman-1.8.0_1
repository: https://alpha.de.repo.voidlinux.org/current
shlib-requires:
	libpthread.so.0
	libgpgme.so.11
	libassuan.so.0
	libgpg-error.so.0
	libseccomp.so.2
	librt.so.1
	libdevmapper.so.1.02
	libc.so.6
short_desc: Simple management tool for containers and images
source-revisions: podman:4351b38206
state: installed

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical box running Void Linux.

Activity

  1. changed the title [-]Rootless podman error: could not get runtime: open /proc/pid/ns/user: no such file or directory[/-] [+]Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory[/+] on Feb 24, 2020
  2. faulesocke commented on Feb 24, 2020

    @faulesocke
    Author

    Oh I just noticed one interesting and probably important thing: The PID in the path (/proc/...) is always the same value: 31678. Of course that PID can not exist. That is probably why the thing is failing.

  3. rhatdan commented on Feb 24, 2020

    @rhatdan
    Contributor

    Could you check if you have a podman running as the user? If yes, could you kill it and then see if the problem goes away?

  4. faulesocke commented on Feb 24, 2020

    @faulesocke
    Author

    No podman running, also tried restarting the box.

  5. rhatdan commented on Feb 24, 2020

    @rhatdan
    Contributor

    What OS are you running podman on? Could this be your kernel does not support User Namespace?
    ls -l /proc/self/ns

  6. faulesocke commented on Feb 24, 2020

    @faulesocke
    Author

    @rhatdan I'm running Void Linux. The Kernel supports user namespaces. The issue is probably not the namespace but podman figuring out some bogus PID when building the path.

    % uname -srvmpio
    Linux 5.4.21_1 #1 SMP PREEMPT Thu Feb 20 09:09:55 UTC 2020 x86_64 unknown unknown GNU/Linux
    
    % cat /etc/os-release 
    NAME="void"
    ID="void"
    DISTRIB_ID="void"
    PRETTY_NAME="void"
    
    % ls -l /proc/self/ns 
    total 0
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 cgroup -> 'cgroup:[4026531835]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 ipc -> 'ipc:[4026531839]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 mnt -> 'mnt:[4026531840]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 net -> 'net:[4026531992]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 pid -> 'pid:[4026531836]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 pid_for_children -> 'pid:[4026531836]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 user -> 'user:[4026531837]'
    lrwxrwxrwx 1 socke socke 0 Feb 24 13:42 uts -> 'uts:[4026531838]'
    

    Could this issue be ulimits related?

  7. giuseppe commented on Feb 24, 2020

    @giuseppe
    Contributor

    could be fixed by: #5237

  8. faulesocke commented on Feb 24, 2020

    @faulesocke
    Author

    @giuseppe Nope. Installed 1.8.1-rc1 and the error is still there.

  9. faulesocke commented on Feb 24, 2020

    @faulesocke
    Author

    I have downgraded a little bit. 1.7.0 is also affected but 1.6.5 works. So this got broken somewhere between 1.6.5 and 1.7.0.

  10. faulesocke commented on Feb 25, 2020

    @faulesocke
    Author

    Okay, this is odd. After I've downgraded it yesterday, i've now upgraded it back to 1.8.0. Then it worked. Then I rebooted to make sure no processes from the old version are running. And it still works. Don't know what went wrong there.

  11. giuseppe commented on Feb 26, 2020

    @giuseppe
    Contributor

    thanks for the update. Let's close the issue for now.

    If it still happens, feel free to re-open it with more details

  12. tunip commented on Feb 26, 2020

    @tunip

    Have this issue too after each reboot of the host system. Only a podman system migrate resolves this temporary (until next boot). Can't downgrade/upgrade the package, as this is a new void linux installation and only podman 1.8.0 packages is available.

  13. rhatdan commented on Feb 27, 2020

    @rhatdan
    Contributor

    Is the runtime directory /run not on a tmpfs? We rely on the /run directory being cleaned on a reboot.

  14. 58 remaining items

  15. Cajga commented on Nov 25, 2020

    @Cajga

    @mheon one more reason why XDG_RUNTIME_DIR is may not been set properly is when one calls podman through a 3rd process like gitlab-runner. This is exactly what we do (using rootless podman from gitlab-runner through a custom executor) and at each call podman receives a new runtime directory (/tmp/custom-executor${RANDOMNUMBER}/run-${UID}) by default (which makes the solution leak podman pause processes for each job).

    I do not know if you considered this in the design but please also note that /run/user/$UID supposed to be deleted when the user logs out from it's last session. From man pam_systemd:

    $XDG_RUNTIME_DIR
    Path to a user-private user-writable directory that is bound to the user login time on the machine. It is automatically created the first time a user logs in and removed on the user's final logout.

    Also, it would be nice to handle cases like sudo -i -u and su -l the same way how normal logins... Probably this is just a dream as pam_systemd is not called and as such no one would handle the directory/variables but who knows :)

  16. mheon commented on Nov 25, 2020

    @mheon
    Contributor
  17. rhatdan commented on Nov 25, 2020

    @rhatdan
    Contributor

    Best you or we can do is handle this on reboot by using tmpfiles.d, if /tmp is not on a tmpfs.
    Podman has no concept of a user logging in or out, and really should not grow this ability.

  18. Cajga commented on Nov 25, 2020

    @Cajga

    @rhatdan IMO that sounds good but only if podman does not rely on the XDG_RUNTIME_DIR variable. As we can see, that is what is causing most of the issues for users in case of rootless podman. It is not set for sudo -i, su -l and also problematic when podman started by systemd and or another process like gitlab-runner (which sets this environment variable to a random directory for each job).
    In case podman would use the "tmpfiles.d interface" of systemd directly, then we could have a fixed path which works in every above mentioned case and the directory cleanup among reboots would be handled by systemd as you said.

  19. rhatdan commented on Nov 25, 2020

    @rhatdan
    Contributor

    Well if the content is always placed on /tmp, then I believe we can setup a tmpfiles.d to look for the directory podman anywhere under /tmp and remove content. We just need to have a path that the tmpfiles.d file would look for.

  20. MICOC commented on Aug 10, 2021

    @MICOC

    Please search conmon.pid in your /runroot/overlay-containers/... folder and delete the file.
    Delete all content in the /tmp/runXXXX folder. Stop Start podman. No reboot is needed.
    Please give feedback if this solution works for you.

  21. dmchandrasinghe commented on Jan 20, 2023

    @dmchandrasinghe

    removing /tmp/runXXXX folder fixed my issue. thanks

  22. locked as resolved and limited conversation to collaborators on Sep 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.rootless

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions