Cannot enter any containers with Shadow 4.19.0: usermod: UID '1000' already exists

[AdamWill]

Describe the bug
Somehow, suddenly, I can't enter any toolbox containers, which is a big problem as I sort of can't do any work. I think this started when I was fiddling with my Fedora 44 container. I entered it, and several tabs which were open in Fedora 43 containers immediately exited (that was weird). Then I tried a dnf update in it, but it hung on the ca-certificates %post. So I decided to delete and re-create it. After a few toolbox rm fedora-toolbox-44, toolbox create fedora-toolbox-44, toolbox enter fedora-toolbox-44 cycles still gave me an old image, I realized I had to do toolbox rmi fedora-toolbox:44 too. Fine. Now I had an up-to-date fedora-toolbox-44. But somewhere along the way I did toolbox enter fedora-toolbox-43 on a different tab and it failed:

Error: usermod: UID '1000' already exists
failed to modify user adamw with UID 1000: failed to invoke usermod(1)

That was worrying, but I figured a reboot would clear it (it usually seems to clear up weird toolbox/podman things). But now I've rebooted, I can't get into fedora-toolbox-43 or fedora-toolbox-44. Verbose logs look basically the same for both:

[adamw@omnibook ~]$ toolbox -v enter fedora-toolbox-44
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Looking up sub-GID and sub-UID ranges for user adamw 
DEBU TOOLBX_DELAY_ENTRY_POINT is                  
DEBU TOOLBX_FAIL_ENTRY_POINT is                   
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbx config directory is /var/home/adamw/.config/toolbox 
DEBU Current Podman version is 5.7.1              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 5.7.1                  
DEBU Migration not needed: Podman version 5.7.1 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/adamw/.config/containers/toolbox.conf not found 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-44'               
DEBU Image: 'fedora-toolbox:44'                   
DEBU Release: '44'                                
DEBU Resolving container and image names          
DEBU Container: 'fedora-toolbox-44'               
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-44'               
DEBU Image: 'fedora-toolbox:44'                   
DEBU Release: '44'                                
DEBU Checking if container fedora-toolbox-44 exists 
DEBU Inspecting container fedora-toolbox-44       
DEBU Entry point of container fedora-toolbox-44 is toolbox (PID=0) 
DEBU Inspecting mounts of container fedora-toolbox-44 
DEBU Generating Container Device Interface for NVIDIA 
DEBU Generating Container Device Interface for NVIDIA: Management Library not found: could not load NVML library: libnvidia-ml.so.1: cannot open shared object file: No such file or directory 
DEBU Generating Container Device Interface for NVIDIA: not a Tegra system: /sys/devices/soc0/family file not found 
DEBU Generating Container Device Interface for NVIDIA: skipping 
DEBU Starting 'p11-kit server' with socket /run/user/1000/toolbox/pkcs11 
DEBU Starting 'p11-kit server': socket /run/user/1000/toolbox/pkcs11 already exists 
DEBU Starting 'p11-kit server': skipping          
DEBU Starting container fedora-toolbox-44         
DEBU Inspecting container fedora-toolbox-44       
DEBU Entry point of container fedora-toolbox-44 is toolbox (PID=0) 
DEBU > Running as real user ID 0                  
DEBU > Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU > TOOLBX_DELAY_ENTRY_POINT is                
DEBU > TOOLBX_FAIL_ENTRY_POINT is                 
DEBU > TOOLBOX_PATH is /usr/bin/toolbox           
DEBU > Migrating to newer Podman                  
DEBU > Migration not needed: running inside a container 
DEBU > Setting up configuration                   
DEBU > Setting up configuration: file /etc/containers/toolbox.conf not found 
DEBU > Setting up configuration: file /root/.config/containers/toolbox.conf not found 
DEBU > Resolving container and image names        
DEBU > Container: ''                              
DEBU > Distribution (CLI): ''                     
DEBU > Image (CLI): ''                            
DEBU > Release (CLI): ''                          
DEBU > Resolved container and image names         
DEBU > Container: 'fedora-toolbox-44'             
DEBU > Image: 'fedora-toolbox:44'                 
DEBU > Release: '44'                              
DEBU > Creating /run/.toolboxenv                  
DEBU > Path /run/host/etc exists                  
DEBU > Resolved /etc/localtime to /run/host/usr/share/zoneinfo/America/Vancouver 
DEBU > Creating regular file /etc/machine-id      
DEBU > Binding /etc/machine-id to /run/host/etc/machine-id 
DEBU > Creating directory /run/systemd/journal    
DEBU > Binding /run/systemd/journal to /run/host/run/systemd/journal 
DEBU > Creating directory /run/systemd/resolve    
DEBU > Binding /run/systemd/resolve to /run/host/run/systemd/resolve 
DEBU > Creating directory /run/systemd/sessions   
DEBU > Binding /run/systemd/sessions to /run/host/run/systemd/sessions 
DEBU > Creating directory /run/systemd/system     
DEBU > Binding /run/systemd/system to /run/host/run/systemd/system 
DEBU > Creating directory /run/systemd/users      
DEBU > Binding /run/systemd/users to /run/host/run/systemd/users 
DEBU > Creating directory /run/udev/data          
DEBU > Binding /run/udev/data to /run/host/run/udev/data 
DEBU > Creating directory /run/udev/tags          
DEBU > Binding /run/udev/tags to /run/host/run/udev/tags 
DEBU > Creating directory /tmp                    
DEBU > Binding /tmp to /run/host/tmp              
DEBU > Creating directory /var/lib/flatpak        
DEBU > Binding /var/lib/flatpak to /run/host/var/lib/flatpak 
DEBU > Creating directory /var/lib/systemd/coredump 
DEBU > Binding /var/lib/systemd/coredump to /run/host/var/lib/systemd/coredump 
DEBU > Creating directory /var/log/journal        
DEBU > Binding /var/log/journal to /run/host/var/log/journal 
DEBU > Creating directory /var/mnt                
DEBU > Binding /var/mnt to /run/host/var/mnt      
DEBU > Creating directory /sys/fs/selinux         
DEBU > Binding /sys/fs/selinux to /usr/share/empty 
DEBU > Preparing to redirect /home to /var/home   
DEBU > /var/home isn't a symbolic link            
DEBU > Redirecting /home to /var/home             
DEBU > Looking up group for sudo                  
DEBU > Group for sudo is wheel                    
DEBU > Modifying user adamw with UID 1000:        
DEBU > usermod                                    
DEBU > --append                                   
DEBU > --groups                                   
DEBU > wheel                                      
DEBU > --home                                     
DEBU > /var/home/adamw                            
DEBU > --password                                 
DEBU >                                            
DEBU > --shell                                    
DEBU > /bin/bash                                  
DEBU > --uid                                      
DEBU > 1000                                       
DEBU > adamw                                      
Error: usermod: UID '1000' already exists
failed to modify user adamw with UID 1000: failed to invoke usermod(1)

Steps how to reproduce the behaviour
Run toolbox enter fedora-toolbox-43 or toolbox enter fedora-toolbox-44

Expected behaviour
A working toolbox

Actual behaviour
An error

Screenshots
Not needed, see output about

Output of toolbox --version (v0.0.90+)
toolbox version 0.3

Toolbx package info (rpm -q toolbox)
toolbox-0.3-2.fc44.x86_64

Output of podman version

Client:        Podman Engine
Version:       5.7.1
API Version:   5.7.1
Go Version:    go1.25.5 X:nodwarf5
Git Commit:    f845d14e941889ba4c071f35233d09b29d363c75
Built:         Tue Dec  9 16:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64

Podman package info (rpm -q podman)
podman-5.7.1-1.fc44.x86_64

Info about your OS
Fedora Silverblue Rawhide

Additional context
N/A (see above)

[debarshiray]

This is odd.

If we look at the state of our daily CI runs on Fedora hosts, linked from the two pills at the bottom of the README.md (I wonder how we ended up with two identical pills with different names), then everything is green.

[debarshiray]

At least on the surface, the container's entry point (ie., the toolbox init-container command) is trying to invoke usermod(8) and failing as:

# usermod \
      --append \
      --groups wheel \
      --home /var/home/adamw \                     
      --password "" \
      --shell /bin/bash \
      --uid 1000 \
      adamw
usermod: UID '1000' already exists

The CI doesn't run on image-based Fedora hosts, only package-based ones. I wonder if this is a difference between the two.

[AdamWill]

The really weird thing is that it didn't apparently start happening in response to any change on the host. The first time I saw the problem (when trying to start an f43 toolbox after they'd all suddenly exited) was on the same boot where previously things were working okay. The problem somehow got worse after rebooting to an updated ostree (now I can't get into f44 containers either), but it didn't start then. It's really weird.

Presumably this usermod is targeted to the new container, right - it's setting up an account that matches my host account?

[AdamWill]

should it be using a subuid here? Not the real uid? Or are subuid / subgid just meant to be used for container-specific file ownerships?

[debarshiray]

The really weird thing is that it didn't apparently start happening in response to any change on the host. The first time I saw the problem (when trying to start an f43 toolbox after they'd all suddenly exited) was on the same boot where previously things were working okay. The problem somehow got worse after rebooting to an updated ostree (now I can't get into f44 containers either), but it didn't start then. It's really weird.

I am setting up a Fedora 43 Silverblue virtual machine to test. Maybe I will need a Rawhide Silverblue to exactly match your environment?

Presumably this usermod is targeted to the new container, right - it's setting up an account that matches my host account?

Yes, exactly.

All non-ancient versions of Podman already set up a user account within the container to match the one on the host. However, it doesn't set all the metadata fields, so Toolbx uses usermod(8) to fill everything in the container's copy of the user account. If the account doesn't exist then it uses useradd(8).

I wonder if there's something funky with the metadata of the host's account that when used with usermod(8) inside the container breaks it.

[debarshiray]

should it be using a subuid here? Not the real uid? Or are subuid / subgid just meant to be used for container-specific file ownerships?

The subordinate IDs are used to create the user namespace for rootless containers.

The container's user namespace can only retain the adamw set of group and user IDs from the host's namespace. The rest of the IDs in the container's namespace come from the subordinate IDs.

If podman create --userns keep-id is used, like Toolbx does, then the adamw host IDs are mapped to the same IDs within the container. In most cases, --userns host is the default and it causes the adamw host IDs to be mapped to 0 within the container.

You can see this in action inside a Toolbx container with:

⬢ [rishi@toolbx ~]$ cat /proc/self/gid_map 
         0          1       1000
      1000          0          1
      1001       1001      64536
⬢ [rishi@toolbx ~]$ cat /proc/self/uid_map 
         0          1       1000
      1000          0          1
      1001       1001      64536

[AdamWill]

Oh, hey. I can see a good explanation of why this is broken with shadow-utils 4.19.0, which almost explains things. It leaves one mystery, though.

The implementation of usermod has changed a lot between 4.18.0 and 4.19.0, but the change that's relevant to us is that it's ditched this bit:

	if (user_newid == user_id) {
		uflg = false;
		oflg = false;
	}

uflg there is its variable to track the value of --uid, basically. So it used to be the case that if you passed --uid as the same value as the user's current UID, after that check, usermod would act as if you had not passed --uid at all.

This is important because of a later bit of code, which hasn't been removed:

	/* local, no need for xgetpwuid */
	if (uflg && !oflg && (prefix_getpwuid (user_newid) != NULL)) {
		fprintf (stderr,
		         _("%s: UID '%lu' already exists\n"),
		         Prog, (unsigned long) user_newid);
		exit (E_UID_IN_USE);
	}

i.e. if uflg is still set when we get here, and the value of it is a UID that already exists, we error out. That's exactly what toolbox is doing, here. So it's not surprising it worked with shadow 4.18.0 but fails with 4.19.0.

The mystery is: why is this happening to my F43 toolbox? That shouldn't have shadow-utils 4.19.0 in it, since that hasn't been built for F43.

However...I was looking at a different bug in shadow yesterday, so it's actually possible I manually installed a 4.19.0 build into my f43 toolbox container. I'll have to see if I can check that somehow.

[debarshiray]

Oh, hey. I can see a good explanation of why this is broken with shadow-utils 4.19.0, which almost explains things. It leaves one mystery, though.

The implementation of usermod has changed a lot between 4.18.0 and 4.19.0, but the change that's relevant to us is that it's ditched this bit:

	if (user_newid == user_id) {
		uflg = false;
		oflg = false;
	}

uflg there is its variable to track the value of --uid, basically. So it used to be the case that if you passed --uid as the same value as the user's current UID, after that check, usermod would act as if you had not passed --uid at all.

This is important because of a later bit of code, which hasn't been removed:

	/* local, no need for xgetpwuid */
	if (uflg && !oflg && (prefix_getpwuid (user_newid) != NULL)) {
		fprintf (stderr,
		         _("%s: UID '%lu' already exists\n"),
		         Prog, (unsigned long) user_newid);
		exit (E_UID_IN_USE);
	}

i.e. if uflg is still set when we get here, and the value of it is a UID that already exists, we error out. That's exactly what toolbox is doing, here. So it's not surprising it worked with shadow 4.18.0 but fails with 4.19.0.

I see. Thanks for doing the Git forensics. :)

I saw that shadow-utils was updated to Shadow 4.19 in Fedora Rawhide on the 12th of January, but I initially ignored it because the upstream Toolbx CI has been passing in the days following that. Now I see that the fedora-toolbox:44 images still have shadow-utils-4.18.0-7.fc44. So the updated Shadow hasn't trickled down to the images.

[AdamWill]

Oh, right, yeah, it's not even in Rawhide yet because it's gated because of the other bug it causes.

So I suspect what happened is I installed a 4.19.0 build into both my f44 and f43 toolbox containers manually in order to test the other bug, and that also caused this breakage :/ If I can find a way to figure out what packages are in the containers I currently can't get into, I'll confirm that...

[debarshiray]

By the way, a combination of podman mount and podman unshare can be useful to debug badly broken rootless containers:

rishi@topinka:~$ podman unshare /bin/bash
root@topinka:~# container_root="$(podman mount fedora-toolbox-44)"
root@topinka:~# ls "$container_root"
afs  boot  etc   lib    media  opt   root  sbin  sys  usr
bin  dev   home  lib64  mnt    proc  run   srv   tmp  var

[AdamWill]

Thanks for the tip! That worked! Managed to downgrade shadow-utils in both f43 and f44 containers and now I can enter them again. Woohoo.