No subuid ranges found for user "<username>" executing any podman command

[praveenkumar]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

Description

$ podman version
ERRO[0000] No subuid ranges found for user "prkumar"

Steps to reproduce the issue:

1. Updated podman from 0.6.4 to 0.7.3 to check non-root user functionality.

$ sudo dnf update podman
[sudo] password for prkumar: 
Fedora 28 - x86_64 - Updates                    8.3 MB/s |  20 MB     00:02    
google-chrome                                    25 kB/s | 3.7 kB     00:00    
RPM Fusion for Fedora 28 - Free - Updates       638 kB/s | 428 kB     00:00    
RPM Fusion for Fedora 28 - Nonfree - Updates    255 kB/s |  83 kB     00:00    
Last metadata expiration check: 0:00:00 ago on Mon 30 Jul 2018 09:49:08 AM IST.
Dependencies resolved.
================================================================================
 Package      Arch         Version                          Repository     Size
================================================================================
Upgrading:
 podman       x86_64       0.7.3-1.git0791210.fc28          updates       6.4 M

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 6.4 M
Is this ok [y/N]: y
Downloading Packages:
podman-0.7.3-1.git0791210.fc28.x86_64.rpm       1.1 MB/s | 6.4 MB     00:06    
--------------------------------------------------------------------------------
Total                                           928 kB/s | 6.4 MB     00:07     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: podman-0.7.3-1.git0791210.fc28.x86_64                  1/1 
  Upgrading        : podman-0.7.3-1.git0791210.fc28.x86_64                  1/2 
  Cleanup          : podman-0.6.4-1.gitd5beb2f.fc28.x86_64                  2/2 
  Running scriptlet: podman-0.6.4-1.gitd5beb2f.fc28.x86_64                  2/2 
  Verifying        : podman-0.7.3-1.git0791210.fc28.x86_64                  1/2 
  Verifying        : podman-0.6.4-1.gitd5beb2f.fc28.x86_64                  2/2 

Upgraded:
  podman.x86_64 0.7.3-1.git0791210.fc28    

2. Executed podman version

Describe the results you received:

ERRO[0000] No subuid ranges found for user "prkumar"

Describe the results you expected:

Version info for podman.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ sudo podman version
[sudo] password for prkumar: 
Version:       0.7.3
Go Version:    go1.10.3
OS/Arch:       linux/amd64

Output of podman info:

$ sudo podman info
host:
  MemFree: 12025827328
  MemTotal: 20731568128
  SwapFree: 10401869824
  SwapTotal: 10401869824
  arch: amd64
  cpus: 4
  hostname: local
  kernel: 4.17.3-200.fc28.x86_64
  os: linux
  uptime: 37h 0m 56.24s (Approximately 1.54 days)
insecure registries:
  registries:
  - 172.30.0.0/16
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 8
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

Can you check the /etc/subuid file to see if their are entries for your user?
grep prkumar /etc/subuid

If this user account has been around for years, then you will have to manually add entries to this file.
useradd does this automatically when creating new users.

grep dwalsh /etc/subuid
dwalsh:100000:65536

[rhatdan]

The error message could be better and does the man page tell you anything?

[rhatdan]

@giuseppe PTAL

[giuseppe]

in the man page we explicitly say the user must be present in the files /etc/subuid and /etc/subgid.

We probably need to make this error message clearer as many users will hit it, it is coming from containers/storage: https://github.com/containers/storage/blob/master/pkg/idtools/idtools.go#L142-L147.

PR here: containers/storage#204

[praveenkumar]

@rhatdan @giuseppe I added the subuids and subgids for the user and they tried again but hit different error now. I also think in the man page if possible to put how to add subuids and subgids if users face issue.

$ sudo usermod --add-subuids 10000-65536 prkumar
$ sudo usermod --add-subgids 10000-65536 prkumar

$ podman version
Version:       0.7.3
Go Version:    go1.10.3
OS/Arch:       linux/amd64

$ podman image pull busybox
Trying to pull docker.io/busybox:latest...Getting image source signatures
Copying blob sha256:75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b
 716.01 KB / 716.01 KB [====================================================] 0s
Copying config sha256:22c2dd5ee85dc01136051800684b0bf30016a3082f97093c806152bf43d4e089
 1.46 KB / 1.46 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
ERRO[0006] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /home: invalid argument 
Failed
Trying to pull registry.fedoraproject.org/busybox:latest...Failed
Trying to pull quay.io/busybox:latest...Failed
Trying to pull registry.access.redhat.com/busybox:latest...Failed
error pulling image "busybox": unable to pull busybox: unable to find image in the registries defined in "/etc/containers/registries.conf"  => this is false error because docker.io is part of registries.conf

$ cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure', 
# and 'registries.block'.

[registries.search]
registries = ['docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.access.redhat.com']

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ["172.30.0.0/16"]


# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []

[giuseppe]

how does your /etc/subuid file look now? I think busybox needs more uids than you currently have mapped

[giuseppe]

@praveenkumar please manually edit your file and add enough user, usermod --add-subuids 10000-65536 USERNAME adds less users as the second argument is the maximum ID of the range, not the range size

[praveenkumar]

@giuseppe after manually updating the range size now able to pull the images.

[giuseppe]

thanks for confirming it. I am going to close this issue as we already are dealing with improving the documentation and the error message

[jschewebbn]

I'm running into this same issue. I've executed the commands to setup my subuid and subgid files and have confirmed that they have my username:10000:65536 in them. My uid is 1832001200.
The output of the pull command above gives the same error

>podman image pull busybox
Trying to pull docker.io/library/busybox...
Getting image source signatures
Copying blob bdbbaa22dec6 done
Copying config 6d5fcfe5ff done
Writing manifest to image destination
Storing signatures
ERRO[0002] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534
for /home): lchown /home: invalid argument
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
Trying to pull quay.io/busybox...
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
Trying to pull registry.fedoraproject.org/busybox...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/busybox...
  name unknown: Repo not found
Error: error pulling image "busybox": unable to pull busybox: 4 errors occurred:
        * Error committing the finished image: error adding layer with blob "sha256:bdbbaa22dec6b7fe23106d2c1b1f43d9598cd8fc33706cc27c1d938ecd5bffc7": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 65534:65534 for /home): lchown /home: invalid argument
        * Error initializing source docker://quay.io/busybox:latest: Error reading manifest latest in quay.io/busybox: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
        * Error initializing source docker://registry.fedoraproject.org/busybox:latest: Error reading manifest latest in registry.fedoraproject.org/busybox: manifest unknown: manifest unknown
        * Error initializing source docker://registry.access.redhat.com/busybox:latest: Error reading manifest latest in registry.access.redhat.com/busybox: name unknown: Repo not found

Version of podman

>podman --version
podman version 1.6.2

[jschewebbn]

Found a solution at #2542.
Executing

rm -rf ~/.{config,local/share}/containers /run/user/$(id -u)/{libpod,runc,vfs-*}

after modifying my sub*uid files fixed it.

[jpescalona]

@jschewebbn You've been my saviour! Works like a charm!

[sistematico]

Found a solution at #2542.
Executing

rm -rf ~/.{config,local/share}/containers /run/user/$(id -u)/{libpod,runc,vfs-*}

after modifying my sub*uid files fixed it.

Awesome! Thank you..

[stemid]

I just have to add my own insights, I came across this thread on Google as I'm sure many others will.

First of all usermod --add-subuid specifies a range, but /etc/subuid shows a starting value and a max value. So usermod --add-subuid 200000-265535 becomes 200000:65536 in subuid file.

Secondly I follow a pattern where I create a user for each container service I want to host, for separation and to avoid root. So I create an elasticsearch user for example, add its subuid and subgid. But this is not enough, all processes of the user must be killed first. In my case there was a lingering podman process belonging to the elasticsearch user even after the users shell had exited.

[mheon]

The podman system migrate command is the recommended way of doing the second bit (clearing open processes to allow Podman to use the new namespace parameters)

[groovyman]

Can you check the /etc/subuid file to see if their are entries for your user?
grep prkumar /etc/subuid

If this user account has been around for years, then you will have to manually add entries to this file.
useradd does this automatically when creating new users.

grep dwalsh /etc/subuid
dwalsh:100000:65536

This was a great help for my problem. After changing the account name (using usermod) the ole account name has been left unchanged in the /etc/subuid file.

[rhatdan]

Might want to open an issue with shadow-utils. This should at least warn you of it.