[intermittent] podman ps --size sometimes returns '0B', 'error unmounting'

[edsantiago]

Seeing this on f29 only; in particular, a non-dnf-upgraded f29. And only intermittently:

# podman run alpine true;podman run alpine dd if=/dev/zero of=/bigfile bs=1M count=10
10+0 records in
10+0 records out

# podman ps --size -a
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/d90aed636ff383f7c3d36367864a87cea6e02e40e51e67913d7df5591553e53e/merged: invalid argument
ERRO[0000] error getting rw size for "25ed230c29166e2df5d6c01da29e4e385619b3378b366ba8a84c1c656b293fc3": error creating overlay mount to /var/lib/containers/storage/overlay/d90aed636ff383f7c3d36367864a87cea6e02e40e51e67913d7df5591553e53e/merged: device or resource busy
CONTAINER ID  IMAGE                            COMMAND               CREATED        STATUS                    PORTS  NAMES             SIZE
25ed230c2916  docker.io/library/alpine:latest  dd if=/dev/zero o...  2 seconds ago  Exited (0) 2 seconds ago         nifty_burnell     0B (virtual 5.58MB)
79415a40fc42  docker.io/library/alpine:latest  true                  3 seconds ago  Exited (0) 3 seconds ago         eloquent_noether  0B (virtual 5.58MB)

It only seems to happen when there are two containers; I haven't yet seen it with only one.

The '0B' thing is sporadic; sometimes it displays 10.5MB.

podman-1.4.4-2.fc29.x86_64; kernel 5.1.16-200.fc29.x86_64

I cannot reproduce on f30. This leads me to suspect that there's a dependency somewhere that needs to be updated. (And if this is the case, the RPM should be rebuilt to Require it). Any ideas on whether this seems plausible, and where I should begin checking?

[rhatdan]

Could it be kernel related? Something to do with the MetaCopy flag?

[edsantiago]

Kernel is latest available on f29, but yes, I suppose it could be metacopy. Let me try removing that from the options.

[edsantiago]

I think that's it. With mountopt = "nodev" I cannot reproduce this, even with automated infinite-loop testing.

If this is the case, I think it might be prudent to package a new containers-common.fc29 without the metacopy option, then build a new podman that Requires: this version.

Very nice thinking, @rhatdan. Thanks for the lead.

[rhatdan]

My concern here is that you are revealing a new bug.

@rhvgoyal WDYT?

[rhvgoyal]

@edsantiago when this happens, you can also capture kernel logs (from dmesg or journal or serial console). May be check journal and there might be some message about why mount failed.

[edsantiago]

Good idea. Log (journalctl) attached. The most prominent diagnostic seems to be:

kernel: overlayfs: lowerdir is in-use as upperdir/workdir

Again, kernel is 5.1.16-200.fc29. On f30, kernel is 5.0.9-301.fc30 but latest available is 5.1.16-300. I'm going to upgrade my f30 and reboot, and see what happens.

podman-metacopy.log

[edsantiago]

Well, well, well. Problem now manifests on f30 with 5.1.16-300.fc30. @rhvgoyal looks like this is headed toward your court. I'll file a kernel bz.

[rhvgoyal]

Latest kernels have added logic to do more rigorous checks to make sure no invalid configuration (overlapping layers are being used). So in this case a directory which is lowerdir is also being used as uppperdir/workdir and that sounds like a problem in conatiner/storage.

[rhvgoyal]

Here is the upstream commit.

commit 146d62e5a5867fbf84490d82455718bfb10fe824
Author: Amir Goldstein amir73il@gmail.com
Date: Thu Apr 18 17:42:08 2019 +0300

ovl: detect overlapping layers

[rhvgoyal]

cc @nalind

Nalin, do you see how this can happen. How a lowerdir is also being used as upperdir/workdir.

[edsantiago]

@rhvgoyal thanks; you've saved me the effort of bisecting and filing a bz. I will stand back now and let (plural) you narrow down the root cause. And, reopening in case it's something fixable in podman.

[rhvgoyal]

cc @amir73il

[cgwalters]

Ah...I started hitting this problem a lot recently, and it being a newer kernel explains a lot.

[cgwalters]

For anyone else like me who would currently choose a functioning container stack and the possibility of (uncharacterized?) undefined behavior over random EBUSY errors during builds:

The upstream patch mentioned above was backported to the Linux stable branch in 5.1.15 as well, so you'll need to revert to an earlier kernel or revert the patch in the 5.1 branch (which is currently used by Fedora) as well.

[edsantiago]

@cgwalters a simpler safer workaround might be to remove ,metacopy=on from mountopt in /etc/containers/storage.conf. AIUI that option is only needed for some advanced rootless cases.

[amir73il]

FYI, the offending kernel patch was also backported to v4.19.56.
I will work on a kernel "fix" to not enforce this with index=off.
Most likely the root cause is container runtime leaking an old overlay mount, whose upper dir
is now attempted to be used in a lower dir stack of a new container, see for reference:
moby/moby#37970
@rhvgoyal I can't find the "proper" docker fix for leaked mounts, but I remember it was discussed.

[rhvgoyal]

@amir73il AFAIK we never fully solved the issue of leaked mounts. We have tried few different things to minimize it. I agree though that this most likely is a result of leaked mount.

For now enforcing error with index=on sounds reasonable. Having said that, that also means that these container runtime will not be able to use index=on till they figure out some proper solution for leaked mount points.

[rhvgoyal]

@amir73il Or we could think of creating a new mount option which does more strict configuration check and enforces errors. (That way people can use index=on without having to worry about leaked mounts).

Otherwise we will just discourage using index=on.

[amir73il]

@rhvgoyal we already have made this compromise about index=on and conflicting upperdir usage. For now my intention is the regain the status quo that my patch has broken.

About discouraging using index=on, I am seeing this as encouraging container runtimes to fix mount leaks ;-)

[amir73il]

Can anyone test the kernel fix above?
It's against current mainline but should apply cleanly to any affected release/stable kernel

[cgwalters]

@cgwalters a simpler safer workaround might be to remove ,metacopy=on from mountopt in /etc/containers/storage.conf. AIUI that option is only needed for some advanced rootless cases.

Tried that

root@orion ~# diff -u /{usr/,}etc/containers/storage.conf
--- /usr/etc/containers/storage.conf	1969-12-31 19:00:00.000000000 -0500
+++ /etc/containers/storage.conf	2019-07-13 13:55:51.947493289 -0400
@@ -33,7 +33,7 @@
 override_kernel_check = "true"
 
 # mountopt specifies comma separated list of extra mount options
-mountopt = "nodev,metacopy=on"
+mountopt = ""
 
 # Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
 # a container, to UIDs/GIDs as they should appear outside of the container, and
root@orion ~# 

but I still see it. Going to try testing the patch.

[cgwalters]

@amir73il I read your patch and it looks sane; trying it out, it works for me as expected. I see in dmesg various:
[ 476.602680] overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
And podman build works (as well as it did before).
Feel free to add Tested-by: Colin Walters <walters@verbum.org>.

[amir73il]

@rhatdan in what way is the kernel message incorrect?
"in-use as upperdir/workdir" means its in use by an overlay mount that has not been unmounted yet - yeh that's what mount leaks would do.
I have posted an RFC patch for a "proper" solution:
https://lore.kernel.org/linux-fsdevel/20190715133839.9878-1-amir73il@gmail.com/
Instead of executing "umount container A", Podman would execute "shutdown fs of container A; umount container A".
The "shutdown fs" operation would release the "in-use" upperdir/workdir and then overlayfs new mounts won't care about the leaked mounts.
If user can access files after fs shutdown through leaked mount in container B they would get EIO.

[rhvgoyal]

@amir73il If I unmount "merged" directory and then delete that directory, then any other leaked mount points on this directory in other mount namespaces go away, I think (As long as other containers are not accessing this directory. If this is unintentional leak, other containers will not be accessing that mount point). I think Eric Biederman made this change. If that's correct, then I am wondering why we don't use this mechanism instead.

[amir73il]

@rhvgoyal I have no objection for using this method. I remember it was suggested at the time of v4.13 kernel regression. Not sure why it wasn't done then...

[rhvgoyal]

@rhatdan can we try this directory removal solution and see if it helps with leaked mount points

[rhvgoyal]

Container removal already cleans up everything. I think for rest of the operations, we probably will have to make sure that after every unmount (once all container/storage reference on merged dir are gone), remove
merged directory and that hopefully takes down all the leaked mount points.

[rhatdan]

@ amir73il the error indicates that one containers upper is being used as the Lower inside of a different container. This is NOT happening. There is no way in the code that we are using the Upper directory in any other container. The upper is being used inside of its own container and is leaked into the second container.
And the kernel is fooled into thinking this is a separate mount of the Upper.

[rhatdan]

@rhvgoyal Would containers/storage#390 fix the issue, or at least mitigate it?

[rhvgoyal]

I think this should help with the issue of unintentionally leaking mount points.

[rhatdan]

@edsantiago Could you verify that this is still an issue with master/podman 1.5.0

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

I believe this is fixed, Reopen if I am mistaken.