Error: container "xxx" is mounted and cannot be removed without using force: container state improper

[hydrargyrum]

Issue Description

some containers started fine, but it seems they can't be cleaned up after stop:

podman run --rm --name=archivebox [some -e options] [some -v options] -it -p 127.0.0.1:5000:8000 docker.io/archivebox/archivebox

then i ctrl-c'ed it, it quit with some errors:

ERRO[304183] cleaning up storage: removing container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8 root filesystem: replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy 
ERRO[304183] Removing container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8: cleaning up container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8: unmounting container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8 storage: cleaning up container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8 storage: unmounting container cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8 root filesystem: replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy 

it cannot be restarted with the same command:

Error: creating container storage: the container name "archivebox" is already in use by cbd1019746f3201ccf55a28b89e2345cbd98e070550f0d922a1cfbeb3b4d22a8. You have to remove that container to be able to reuse that name: that name is already in use by an external entity, or use --replace to instruct Podman to do so.

but:

% podman stop archivebox
Error: no container with name or ID "archivebox" found: no such container
% podman rm archivebox  
Error: container "archivebox" is mounted and cannot be removed without using force: container state improper

adding --replace doesn't help:

WARN[0000] Unmounting container "archivebox" while attempting to delete storage: replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy 
Error: removing storage for container "archivebox": replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy

i have the same situation with multiple containers: started fine, and now cannot be stopped properly.
i have to add some context which may be related:

1. containers used to start/stop properly and were running when…
2. one day, the system was hard-rebooted (for a reason unrelated to podman)
3. and now, those containers were able to start but not really to stop

i don't mind if the containers are lost but i don't want to lose the volumes

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

Client:        Podman Engine
Version:       5.4.2
API Version:   5.4.2
Go Version:    go1.24.2
Built:         Fri May  9 02:43:40 2025
Build Origin:  Debian
OS/Arch:       linux/amd64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mheon]

Can you try adding --force to podman rm? From the manpage:

Force the removal of running and paused containers. Forcing a container removal also removes containers from container storage even if the container is not known to Podman.  For example, containers that are created by different container engines like Buildah.  In addition, forcing can be used to remove unusable containers, e.g. containers whose OCI runtime has become unavailable.

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[hydrargyrum]

WARN[0000] Unmounting container "archivebox" while attempting to delete storage: replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy 
Error: removing storage for container "archivebox": replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy

can't be started even after this

[mheon]

Root or rootless Podman?

If root, can you do a mount | grep /data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged and provide the output?

[hydrargyrum]

it's rootless, so i took the liberty to run it inside podman unshare (not sure if it's the right thing):

overlay on /data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged type overlay (rw,relatime,lowerdir=/data/podman/overlay/l/BLRK47JAHV4CVV3NQLX5VCFBS2:/data/podman/overlay/l/P7BK2ZIRUIQKD6WI2ZFMDMWAXK:/data/podman/overlay/l/SS4K7OF5DHWV7PZUKJIZVHRIO6:/data/podman/overlay/l/MKFCHKY3N4ILPKEB3YWPTUMTDB:/data/podman/overlay/l/R4XPTTNKGBDLVEP73MXY5X72QT:/data/podman/overlay/l/3AWOCVFNIPUBNYOQ7AHG46LGY7:/data/podman/overlay/l/ERMEBBC6S6PWJL4QYJQQ35PI6J:/data/podman/overlay/l/MMNCMJ3CQ5STKRUBSWGKHAY5AC:/data/podman/overlay/l/PUBIJV4RNXHZTPN6N5FJR4JNWR:/data/podman/overlay/l/QQXOKQGELBAL3L45KS23DQI2IQ:/data/podman/overlay/l/ZNWDVUGNN7TZGT6KSYCOFJJUKT:/data/podman/overlay/l/NCDYU77DYA6WGJF62PJEGH2ISB:/data/podman/overlay/l/C6SMQ2B6SPOBXAQCL5ZZVRXFPB:/data/podman/overlay/l/5N6PV6QZ6WBBESLKX3VFC2I3QZ:/data/podman/overlay/l/73DRK5XUHFTYYHJBMC6EDGQKI3:/data/podman/overlay/l/IVXLTI4XV5HSL6MVFGRAIQUWKE:/data/podman/overlay/l/A5RVWU6HMXBI4JUMUN2T5WVNDK:/data/podman/overlay/l/P62G2SGIYHWVRCYKCWAS4GXILX:/data/podman/overlay/l/HCKK5CLM7L4A63VVNERT4ZAE3F:/data/podman/overlay/l/NM7KXLNSVPNLORBN3M7RR3VMYU:/data/podman/overlay/l/MCXM3WEVGLHQHXFCVLF6OWBDSA:/data/podman/overlay/l/P65XPQ3SDE2RONCP6ZGAG4N3BC:/data/podman/overlay/l/GGOC7JUEWLK5I7FGBKX7XV3G6Q:/data/podman/overlay/l/BXFGVVZRQSP5HDCKQVVTKEJ7HV:/data/podman/overlay/l/5AD4SNZOEMFCXUFFXKOQMYYNZP:/data/podman/overlay/l/MIJGDIG2O57YZHP73LZEV4XHK4,upperdir=/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/diff,workdir=/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/work,redirect_dir=nofollow,uuid=on,volatile,userxattr)

[mheon]

I assume a manual unmount doesn't work? Do you see anything under that path in lsof? Both within podman unshare to get inside the namespace

[hydrargyrum]

umount /data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged exits with status 32 and message not mounted though it still appears in mount output.
lsof | grep 069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784 doesn't show anything.

[hydrargyrum]

ran podman system check and regarding this hash, got

Damaged layer 069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784:
1 error occurred:
        * layer 069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784: replacing mount point "/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged": device or resource busy

got this device or resource busy error for other layers as well

also not sure how much bad they are, but got other errors regarding other layers like:

Damaged layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d:
5 errors occurred:
        * layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d: -/dev/random, layer content modified
        * layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d: -/dev/null, layer content modified
        * layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d: -/dev/console, layer content modified
        * layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d: -/dev/zero, layer content modified
        * layer c4881a985315a082795e61811ba8d3fbf2fc3c297d24bc35a99c7fd159f1a09d: -/dev/urandom, layer content modified

[hydrargyrum]

How to investigate more or solve this?

[mheon]

If mount fails to unmount with not mounted, but the mount is still visible in mount, I feel like this has surpassed Podman and become a kernel issue. @Luap99 Agree?

[Luap99]

@hydrargyrum Did you reboot the system and can reliable reproduce this issue afterwards?

I would not say it is a kernel issue necessarily, the way bind mounts and mount propagation are handled is extremely confusing and can lead to such situations. Because it is possible to mount /dir/abc and then bind mount /dir to /dir (i.e. to force a private mount progation) and then the umount will fail with EBUSY. Note that umount command will always claim not mounted not sure why.

To show what I mean

# create new mount namespace first
$ unshare -rm

$ mkdir -p /tmp/test/tmp
$ mount -t tmpfs none /tmp/test/tmp
$ mount --bind /tmp/test /tmp/test
$ findmnt -R /tmp
TARGET          SOURCE       FSTYPE OPTIONS
/tmp            tmpfs        tmpfs  rw,nosuid,nodev,seclabel,nr_inodes=1048576,inode64
├─/tmp/test/tmp none         tmpfs  rw,relatime,seclabel,uid=1000,gid=1000,inode64
└─/tmp/test     tmpfs[/test] tmpfs  rw,nosuid,nodev,seclabel,nr_inodes=1048576,inode64
$ umount /tmp/test/tmp 
umount: /tmp/test/tmp: not mounted.

# the only way to umount now is to first unmount the bind mount again
$ umount /tmp/test
$ umount /tmp/test/tmp 

So what is the output of podman unshare findmnt when the umount issue happens?

[hydrargyrum]

Did you reboot the system and can reliable reproduce this issue afterwards?

didn't try to reboot yet, should i?

about findmnt, only one line matches 069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784:

/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/merged                  overlay                              overlay         rw,relatime,lowerdir=/data/podman/overlay/l/BLRK47JAHV4CVV3NQLX5VCFBS2:/data/podman/overlay/l/P7BK2ZIRUIQKD6WI2ZFMDMWAXK:/data/podman/overlay/l/SS4K7OF5DHWV7PZUKJIZVHRIO6:/data/podman/overlay/l/MKFCHKY3N4ILPKEB3YWPTUMTDB:/data/podman/overlay/l/R4XPTTNKGBDLVEP73MXY5X72QT:/data/podman/overlay/l/3AWOCVFNIPUBNYOQ7AHG46LGY7:/data/podman/overlay/l/ERMEBBC6S6PWJL4QYJQQ35PI6J:/data/podman/overlay/l/MMNCMJ3CQ5STKRUBSWGKHAY5AC:/data/podman/overlay/l/PUBIJV4RNXHZTPN6N5FJR4JNWR:/data/podman/overlay/l/QQXOKQGELBAL3L45KS23DQI2IQ:/data/podman/overlay/l/ZNWDVUGNN7TZGT6KSYCOFJJUKT:/data/podman/overlay/l/NCDYU77DYA6WGJF62PJEGH2ISB:/data/podman/overlay/l/C6SMQ2B6SPOBXAQCL5ZZVRXFPB:/data/podman/overlay/l/5N6PV6QZ6WBBESLKX3VFC2I3QZ:/data/podman/overlay/l/73DRK5XUHFTYYHJBMC6EDGQKI3:/data/podman/overlay/l/IVXLTI4XV5HSL6MVFGRAIQUWKE:/data/podman/overlay/l/A5RVWU6HMXBI4JUMUN2T5WVNDK:/data/podman/overlay/l/P62G2SGIYHWVRCYKCWAS4GXILX:/data/podman/overlay/l/HCKK5CLM7L4A63VVNERT4ZAE3F:/data/podman/overlay/l/NM7KXLNSVPNLORBN3M7RR3VMYU:/data/podman/overlay/l/MCXM3WEVGLHQHXFCVLF6OWBDSA:/data/podman/overlay/l/P65XPQ3SDE2RONCP6ZGAG4N3BC:/data/podman/overlay/l/GGOC7JUEWLK5I7FGBKX7XV3G6Q:/data/podman/overlay/l/BXFGVVZRQSP5HDCKQVVTKEJ7HV:/data/podman/overlay/l/5AD4SNZOEMFCXUFFXKOQMYYNZP:/data/podman/overlay/l/MIJGDIG2O57YZHP73LZEV4XHK4,upperdir=/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/diff,workdir=/data/podman/overlay/069e0588bb42c6e74b8df8d0ad1e3f5faa00fa635b4a6be3acf8aa2429069784/work,redirect_dir=nofollow,uuid=on,volatile,userxattr