Creating diff of a container running on overlayfs generates duplicate mounts

[foamkeen]

Description

When we try to create a diff of a container running on overlayfs with ctr utility, we immediately see the following in the kernel log:

kernel: [42256.430385] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
kernel: [42256.430389] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

And later for that container we observe this "undefined behavior" as the RootFS of that container becoming partially not-writable (an attempt to create a file results in "No such file or directory" error).

My understanding is that this happens because Compare function of walkingDiff does temporary mounts:

containerd/diff/walking/differ.go

Line 89 in 45e0e5a

if err := mount.WithTempMount(ctx, lower, func(lowerRoot string) error {

And it does it for an 'Active' snapshot using the same workdir and upperdir, but for overlayfs you are not supposed to do this.

As a solution I see a conversion of a mount to a read-only one before passing it to Compare. It can be done by altering the mount options like so: workdir and upperdir options are removed and a path from upperdir option is moved to the leftmost in lowerdir option.

Steps to reproduce the issue

1. Run ctr snapshot diff <container id> for a container running on overlayfs and observe overlayfs complains about upperdir and workdir in the kernel log.

Describe the results you received and expected

Running ctr snapshot diff on overlayfs should not corrupt the RootFS of a running container.

What version of containerd are you using?

containerd github.com/containerd/containerd 1.4.8+azure 7eba593

Any other relevant information

runc --version:

runc version 1.0.0-rc95
commit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.5.1

uname -a:

Linux aks-nodepool1-24028517-vmss000002 5.4.0-1056-azure #58~18.04.1-Ubuntu SMP Wed Jul 28 23:14:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

[Bryce-huang]

Any update for this issue? This is a big problem!

[AkihiroSuda]

Fixed in #8259