overlay2 + linux v4.13: error creating overlay mount to /var/lib/docker/overlay2/ID/merged: device or resource busy #34672
Comments
|
I can still reproduce this with the 4.13.2 kernel, so it wasn't just the rc. The below has reproduced it for me fairly reliably. The number 100 is rather arbitrary. 2 was too few, 100 is probably more than is strictly needed. ==> make_sleep_container.sh <==
#!/bin/bash
tmpdir=$(mktemp -d)
cd $tmpdir
echo -e "FROM scratch\nCOPY . /" > Dockerfile
b=$(which sleep)
libs=$(sudo ldd $b | grep -o /lib'[^ ]*' | sort -u)
sudo rsync -av --relative --copy-links $b $libs ./
sudo docker build -t sleep .
==> repro.sh <==
#!/bin/bash
set -ex
sudo systemctl stop docker
sudo rm -rf /var/lib/docker
systemctl restart docker
./make_sleep_container.sh
pids=""
for i in $(seq 1 100); do
docker run --rm --net=none sleep sleep 0.2 &
pids="$pids $!"
done
wait $pidsRunning
For me, this happens within the first couple runs. |
|
These errors always follow a 'Failed to unmount' message. From the daemon's debug logs:
|
|
ping @dmcgowan |
|
@euank thanks for reporting, I haven't seen these |
|
I've dug into it more and I think the The On a hunch, I removed the At this point I suspect that this is a kernel bug. My next step is to try and write a reproduction that doesn't involve dockerd. |
|
Thanks for the update @euank |
|
Edit: this comment is heading down the wrong path, none of the below is particularly relevant After more investigation: Using perf probes, we get that the kernel
The important detail to notice here is that Because of that, umount returning isn't enough for us to be sure the upperdir is unlocked from overlayfs' view. The easiest fix would be to retry a few times on The most confusing thing to me right now is that I'm still unable to figure out a simple reproduction without docker. I'm not sure what I'm missing there. |
|
Retrying on ebusy shows me that it takes from around 200ms to 1200ms for the That seems like an excessively long amount of time for the explanation I've currently got. |
|
After a bit more adventuring, I believe I've tracked this down to two distinct problems which both cause this issue identically! First, a helpful trick for reproducing this -- add a 1-3 second sleep before pivoting root: And a few hundred ms in the moby/daemon/graphdriver/overlay2/overlay.go Line 610 in ba31763 After those changes, it reliably reproduces running just two containers at once, which made it much easier to continue investigating. Anyways, it turns out that if you look through every mount namespace for references to the overlayfs mount, you'll find that the This copy is a private copy of the mount meaning our host umount won't get it, and we're at the mercy of this other container's I've created a commented reproduction of what docker and runc are doing namespace and mount wise leading up to this ebusy: #!/bin/bash
set -x
# c1 and c2 represent two different docker containers starting at once
c1=1
c2=2
function ovlOpts() {
echo -n "lowerdir=$tmpdir/lower,upperdir=$tmpdir/$1/diff,workdir=$tmpdir/$1/work"
}
tmpdir=$(mktemp -d) # 'overlay2' graphdriver dir
mkdir -p $tmpdir/{$c1,$c2}/{diff,merged,work}
mkdir -p $tmpdir/lower
# overlay2 driver in its setup code does this
# https://github.com/moby/moby/blob/ba317637de9b9918cdc2139466dd51c6200bd158/daemon/graphdriver/overlay2/overlay.go#L178
mount --bind $tmpdir $tmpdir
mount --make-private $tmpdir
# Container 2 sets up
# https://github.com/moby/moby/blob/ba317637de9b9918cdc2139466dd51c6200bd158/daemon/graphdriver/overlay2/overlay.go#L589
mount -t overlay overlay -o "$(ovlOpts $c2)" $tmpdir/$c2/merged
# Container 1 starts setting up
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged
# Container 2 runs 'runc init' code in parallel
(
# https://github.com/opencontainers/runc/blob/8b47a242a9aebdfe1c0c2b6513368f736d505bf0/libcontainer/nsenter/nsexec.c#L823
unshare -m --propagation unchanged -- bash <<EOF
# Now runc init remounts /
# https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605
# Due to how the config conversion works, config.RootPropagation is never 0,
# and defaults instead to MS_PRIVATE | MS_REC. I'll PR a fix
mount --make-rprivate /
# Now a bunch of init stuff happens, including premount cmds and hooks
sleep 1
# .. and then pivot root happens which cleans up our old root
# It's hard to do in shell, so we'll just pretend an umount of / is close enough
# https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L676
cd /
umount -l .
EOF
) &
sleep 0.5
# While container2 is doing its init, container 1 unmounts and remounts its overlay
umount $tmpdir/$c1/merged
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged
# Boom, EBUSY on 4.13+ because `unshare -m` above has a private copy of the mount
sleep 1
# Now that the runc init code has pivoted and umounted its old root, we're able to mount without EBUSY
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged
umount $tmpdir/$c1/merged
# Cleanup
umount $tmpdir/$c2/merged
umount $tmpdir
rm -rf $tmpdirChanging the runc-init mount to be I'll PR each of those changes shortly with suitable commit messages. |
The code in prepareRoot (https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605) attempts to default the rootfs mount to `rslave`. However, since the spec conversion has already defaulted it to `rprivate`, that code doesn't actually ever do anything. This changes the spec conversion code to accept "" and treat it as 0. Implicitly, this makes rootfs propagation default to `rslave`, which is a part of fixing the moby bug moby/moby#34672 Alternate implementatoins include changing this defaulting to be `rslave` and removing the defaulting code in prepareRoot, or skipping the mapping entirely for "", but I think this change is the cleanest of those options.
|
Thanks for the extensive analysis @euank |
The code in prepareRoot (https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605) attempts to default the rootfs mount to `rslave`. However, since the spec conversion has already defaulted it to `rprivate`, that code doesn't actually ever do anything. This changes the spec conversion code to accept "" and treat it as 0. Implicitly, this makes rootfs propagation default to `rslave`, which is a part of fixing the moby bug moby/moby#34672 Alternate implementatoins include changing this defaulting to be `rslave` and removing the defaulting code in prepareRoot, or skipping the mapping entirely for "", but I think this change is the cleanest of those options. Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
This reverts commit e076bcc. It also reverts it for the overlay2 package, which didn't exist at the time the commit was made but is a direct successor with copy-pasted code. The original commit was *meant* to fix a bug whereby `docker cp` (implemented via chrootarchive) could inadvertantly lead to shared mounts getting unmounted on the host too. The fix, however, is wrong. It results in unmounting overlay mounts reliably, in the face of multiple mount namespaces, being impossible. This hasn't been noticed until now because on kernels prior to v4.13, temporarily leaking overlayfs mounts to other namespaces didn't have any ill effects. Starting with v4.13, setting the mount to private and thus leaking mounts results in errors. See moby#34672 The correct fix for the original issue was implemented later in moby#27609, and since that code is now merged we can safely throw away this incorrect fix. Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
This reverts commit e076bcc. It also reverts it for the overlay2 package, which didn't exist at the time the commit was made but is a direct successor with copy-pasted code. The original commit was *meant* to fix a bug whereby `docker cp` (implemented via chrootarchive) could inadvertantly lead to shared mounts getting unmounted on the host too. The fix, however, is wrong. It results in unmounting overlay mounts reliably, in the face of multiple mount namespaces, being impossible. This hasn't been noticed until now because on kernels prior to v4.13, temporarily leaking overlayfs mounts to other namespaces didn't have any ill effects. Starting with v4.13, setting the mount to private and thus leaking mounts results in errors. See moby#34672 The correct fix for the original issue was implemented later in moby#27609, and since that code is now merged we can safely throw away this incorrect fix. Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
|
In my testing, this is fixed by applying both #34948 and opencontainers/runc#1598 One or the other isn't enough since both result in a private copy of the mount off in runc's init mountns, but both of them together ensure it doesn't happen. |
This reverts commit e076bcc. It also reverts it for the overlay2 package, which didn't exist at the time the commit was made but is a direct successor with copy-pasted code. The original commit was meant to fix a bug whereby `docker cp` (implemented via chrootarchive) could inadvertantly lead to shared mounts getting unmounted on the host too. The fix, however, had side effects. It results in overlay mounts being private, and thus being quite easy to leak copies that are hard to umount into other mount namespaces on the box. This hasn't been noticed until now because on kernels prior to v4.13, temporarily leaking overlayfs mounts to other namespaces didn't have any ill effects. Starting with v4.13, setting the mount to private and thus leaking mounts results in errors. See moby#34672 The correct fix for the original issue was implemented later in moby#27609, and since that code is now merged we can safely throw away this less ideal fix. Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
|
I think it's still possible to hit the race in theory without also applying opencontainers/runc#1500, but in practice I can't hit that one. |
See coreos/bugs#2127 and moby/moby#34672 for discussion. Patch files have been split into more folders, plus some manual eapply calls, to allow moby/moby patches to be used unmodified against docker-ce
|
This looks like another variant of #34573 (effectively mount leaks are causing |
See coreos/bugs#2127 and moby/moby#34672 for discussion. Patch files have been split into more folders, plus some manual eapply calls, to allow moby/moby patches to be used unmodified against docker-ce
See coreos/bugs#2127 and moby/moby#34672 for discussion. Patch files have been split into more folders, plus some manual eapply calls, to allow moby/moby patches to be used unmodified against docker-ce
|
Hi, I think I can confirm this behaviour as of kernel 4.13.3 and docker-ce 17.09! docker version:
dmesg:
docker info:
|
|
Will upgrade the kernel version and check |
Enforcing exclusive ownership on upper/work dirs caused a docker regression: moby/moby#34672. Euan spotted the regression and pointed to the offending commit. Vivek has brought the regression to my attention and provided this reproducer: Terminal 1: mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none merged/ Terminal 2: unshare -m Terminal 1: umount merged mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none merged/ mount: /root/overlay-testing/merged: none already mounted or mount point busy To fix the regression, I replaced the error with an alarming warning. With index feature enabled, mount does fail, but logs a suggestion to override exclusive dir protection by disabling index. Note that index=off mount does take the inuse locks, so a concurrent index=off will issue the warning and a concurrent index=on mount will fail. Documentation was updated to reflect this change. Fixes: 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") Cc: <stable@vger.kernel.org> # v4.13 Reported-by: Euan Kemp <euank@euank.com> Reported-by: Vivek Goyal <vgoyal@redhat.com> Signed-off-by: Amir Goldstein <amir73il@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Enforcing exclusive ownership on upper/work dirs caused a docker regression: moby/moby#34672. Euan spotted the regression and pointed to the offending commit. Vivek has brought the regression to my attention and provided this reproducer: Terminal 1: mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none merged/ Terminal 2: unshare -m Terminal 1: umount merged mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none merged/ mount: /root/overlay-testing/merged: none already mounted or mount point busy To fix the regression, I replaced the error with an alarming warning. With index feature enabled, mount does fail, but logs a suggestion to override exclusive dir protection by disabling index. Note that index=off mount does take the inuse locks, so a concurrent index=off will issue the warning and a concurrent index=on mount will fail. Documentation was updated to reflect this change. Fixes: 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") Cc: <stable@vger.kernel.org> # v4.13 Reported-by: Euan Kemp <euank@euank.com> Reported-by: Vivek Goyal <vgoyal@redhat.com> Signed-off-by: Amir Goldstein <amir73il@gmail.com> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Same process to get in latest bpftool and iproute2 changes. For net-next, the build 28 went fine, but CI is hitting the docker bug in moby/moby#34672, leave it out for now. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Same process to get in latest bpftool and iproute2 changes. For net-next, the build 28 went fine, but CI is hitting the docker bug in moby/moby#34672, leave it out for now. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e5a586 ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Source: Kernel.org MR: 100044 Type: Integration Disposition: Backport from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable linux-4.19.y ChangeID: 9c13e770cca978b8d5c870f2270cdf3a10c63817 Description: commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e5a586 ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster@mvista.com>
commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
BugLink: https://bugs.launchpad.net/bugs/1845642 commit 0be0bfd upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e5a586 ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
BugLink: https://bugs.launchpad.net/bugs/1846277 commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e5a586 ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kamal Mostafa <kamal@canonical.com> Signed-off-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream. Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs") in v4.13 added some sanity checks on overlayfs layers. This change caused a docker regression. The root cause was mount leaks by docker, which as far as I know, still exist. To mitigate the regression, commit 85fdee1 ("ovl: fix regression caused by exclusive upper/work dir protection") in v4.14 turned the mount errors into warnings for the default index=off configuration. Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in v5.2, re-introduced exclusive upper/work dir checks regardless of index=off configuration. This changes the status quo and mount leak related bug reports have started to re-surface. Restore the status quo to fix the regressions. To clarify, index=off does NOT relax overlapping layers check for this ovelayfs mount. index=off only relaxes exclusive upper/work dir checks with another overlayfs mount. To cover the part of overlapping layers detection that used the exclusive upper/work dir checks to detect overlap with self upper/work dir, add a trap also on the work base dir. Link: moby/moby#34672 Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/ Link: containers/podman#3540 Fixes: 146d62e5a586 ("ovl: detect overlapping layers") Cc: <stable@vger.kernel.org> # v4.19+ Signed-off-by: Amir Goldstein <amir73il@gmail.com> Tested-by: Colin Walters <walters@verbum.org> Signed-off-by: Miklos Szeredi <mszeredi@redhat.com> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com> Acked-by: Joseph Qi <joseph.qi@linux.alibaba.com>
|
$ vim /etc/docker/daemon.json I had the same issue with the docker. It was not able to push images to GCR. The above will able to solved my issue. |
Description
The overlay driver in the kernel, starting with 4.13, will return an error for overlay mounts that re-use the upper dir. This error was introduced in this patch.
Using docker
17.06.1-ceon the4.13-rc6kernel I can unreliably reproduce this error message.I've only ever observed it on the first container run, and only infrequently. I assume that there are two mounts that race and sometimes clash.
Steps to reproduce the issue:
/var/lib/dockerdirectory. Start dockerd, and as soon as possible, run a few dozen containers in parallel.Note that this only impacts running multiple containers at once. Serializing all container runs avoids it.
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
I've seen it on AWS and Qemu, presumably happens on all.
I've also reported this issue over here on the CoreOS bug tracker.
The text was updated successfully, but these errors were encountered: