Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overlay2 + linux v4.13: error creating overlay mount to /var/lib/docker/overlay2/ID/merged: device or resource busy #34672

Closed
euank opened this issue Aug 29, 2017 · 37 comments

Comments

@euank
Copy link
Contributor

@euank euank commented Aug 29, 2017

Description

The overlay driver in the kernel, starting with 4.13, will return an error for overlay mounts that re-use the upper dir. This error was introduced in this patch.

Using docker 17.06.1-ce on the 4.13-rc6 kernel I can unreliably reproduce this error message.
I've only ever observed it on the first container run, and only infrequently. I assume that there are two mounts that race and sometimes clash.

Steps to reproduce the issue:

  1. Install the 4.13 kernel
  2. Boot the machine with an empty /var/lib/docker directory. Start dockerd, and as soon as possible, run a few dozen containers in parallel.
  3. Occasionally, (perhaps 1 out of 30 runs), get the error "error creating overlay mount to /var/lib/docker/overlay2/ID/merged: device or resource busy"
  4. Note that the dmesg output includes "overlayfs: upperdir is in-use by another mount"

Note that this only impacts running multiple containers at once. Serializing all container runs avoids it.

Output of docker version:

$ docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.2
 Git commit:   874a737
 Built:        Sat Aug 26 01:07:04 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.2
 Git commit:   874a737
 Built:        Fri Aug 25 18:06:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.13.0-rc6-coreos
Operating System: Container Linux by CoreOS 1506.0.0+2017-08-25-1813 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.3MiB
Name: localhost
ID: ZBQY:PD55:UTX2:K2N4:CPQJ:HWIY:SOIQ:IC6P:NNXT:YKUZ:XFNP:ESWJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

I've seen it on AWS and Qemu, presumably happens on all.

I've also reported this issue over here on the CoreOS bug tracker.

@euank
Copy link
Contributor Author

@euank euank commented Sep 19, 2017

I can still reproduce this with the 4.13.2 kernel, so it wasn't just the rc.

The below has reproduced it for me fairly reliably. The number 100 is rather arbitrary. 2 was too few, 100 is probably more than is strictly needed.

==> make_sleep_container.sh <==
#!/bin/bash

tmpdir=$(mktemp -d)
cd $tmpdir

echo -e "FROM scratch\nCOPY . /" > Dockerfile
b=$(which sleep)
libs=$(sudo ldd $b | grep -o /lib'[^ ]*' | sort -u)
sudo rsync -av --relative --copy-links $b $libs ./
sudo docker build -t sleep .




==> repro.sh <==
#!/bin/bash

set -ex

sudo systemctl stop docker
sudo rm -rf /var/lib/docker
systemctl restart docker

./make_sleep_container.sh

pids=""

for i in $(seq 1 100); do
  docker run --rm --net=none sleep sleep 0.2 &
  pids="$pids $!"
done

wait $pids

Running repro.sh on the 4.13 kernel with 17.06.2-ce gives me:

/run/torcx/bin/docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/cca2481a00020458e95d589fca23192c1addc1dfeab6793d503e1749a2acbef6/merged: device or resource busy.
ERRO[0008] error waiting for container: context canceled 
/run/torcx/bin/docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/14464288a0c06dfead5dcbec623037ab0e29dbd3b35cf4bc439865af475c44d7/merged: device or resource busy.
ERRO[0008] error waiting for container: context canceled 
/run/torcx/bin/docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/f0fa08260045a14087761dc2bd7bd6b3c82c656814630f0d996b67849cca9e41/merged: device or resource busy.
ERRO[0008] error waiting for container: context canceled 

For me, this happens within the first couple runs.

@euank
Copy link
Contributor Author

@euank euank commented Sep 19, 2017

These errors always follow a 'Failed to unmount' message. From the daemon's debug logs:

$ sudo journalctl -u docker -o cat | grep -E "level=error|Failed" 
....

time="2017-09-19T22:29:07.223235635Z" level=debug msg="Failed to unmount 62676b62ee42ac87c68d82092051e683037dfaa4dfb016fb4a30273d1e12e4ae overlay: /var/lib/docker/overlay2/62676b62ee42ac87c68d82092051e683037dfaa4dfb016fb4a30273d1e12e4ae/merged - invalid argument"
time="2017-09-19T22:29:08.277011559Z" level=error msg="Handler for POST /v1.30/containers/2cfb5c61a0713b6b163e3a2abfb4572dfcf6a167c7837b58ad217870bdc7b661/start returned error: error creating overlay mount to /var/lib/docker/overlay2/62676b62ee42ac87c68d82092051e683037dfaa4dfb016fb4a30273d1e12e4ae/merged: device or resource busy"
time="2017-09-19T22:29:08.331422149Z" level=debug msg="Failed to unmount e5df5519f398a1c1f43969662c7872f75cc2432f02c67f7a3164babd21ada536 overlay: /var/lib/docker/overlay2/e5df5519f398a1c1f43969662c7872f75cc2432f02c67f7a3164babd21ada536/merged - invalid argument"
time="2017-09-19T22:29:08.364760182Z" level=error msg="Handler for POST /v1.30/containers/dffd9bcd3a7da4716fd445b40d48f87ee04c6361d079e71750e9d66795ca2f7e/start returned error: error creating overlay mount to /var/lib/docker/overlay2/e5df5519f398a1c1f43969662c7872f75cc2432f02c67f7a3164babd21ada536/merged: device or resource busy"
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 20, 2017

ping @dmcgowan

@dmcgowan
Copy link
Member

@dmcgowan dmcgowan commented Sep 20, 2017

@euank thanks for reporting, I haven't seen these Failed to unmount 62676b62ee42ac87c68d82092051e683037dfaa4dfb016fb4a30273d1e12e4ae overlay: /var/lib/docker/overlay2/62676b62ee42ac87c68d82092051e683037dfaa4dfb016fb4a30273d1e12e4ae/merged - invalid argument yet, but the combination of the kernel change and this failure makes sense.

@euank
Copy link
Contributor Author

@euank euank commented Sep 20, 2017

I've dug into it more and I think the invalid argument messages are actually red herrings.

The EBUSY is what actually matters. With some stracing and added logging, it appears to me that the overlay2 driver's locking mechanism is working just fine.
I see the EBUSY on a mount call for a given directory even though a umount call to it had returned 0.

On a hunch, I removed the MNT_DETACH flag, but that didn't make a difference.

At this point I suspect that this is a kernel bug. My next step is to try and write a reproduction that doesn't involve dockerd.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 20, 2017

Thanks for the update @euank

@euank
Copy link
Contributor Author

@euank euank commented Sep 21, 2017

Edit: this comment is heading down the wrong path, none of the below is particularly relevant

After more investigation:

Using perf probes, we get that the kernel ovl_inuse_unlock happens after this chain:

task_work_run
__cleanup_mnt
cleanup_mnt
deactivate_super
deactivate_locked_super
kill_anon_super
generic_shutdown_super
ovl_inuse_unlock

The important detail to notice here is that ovl_inuse_unlock is coming from a task queue (task_work_run).

Because of that, umount returning isn't enough for us to be sure the upperdir is unlocked from overlayfs' view.
Furthermore, there doesn't appear to be any way to actually check whether the upperdir is unlocked since the mountpoint has already been cleaned up.

The easiest fix would be to retry a few times on EBUSY in the overlay2 mount code.

The most confusing thing to me right now is that I'm still unable to figure out a simple reproduction without docker. I'm not sure what I'm missing there.

@euank
Copy link
Contributor Author

@euank euank commented Sep 21, 2017

Retrying on ebusy shows me that it takes from around 200ms to 1200ms for the mount to actually succeed.

That seems like an excessively long amount of time for the explanation I've currently got.
That makes me suspect there's something more at play.

@euank
Copy link
Contributor Author

@euank euank commented Sep 22, 2017

After a bit more adventuring, I believe I've tracked this down to two distinct problems which both cause this issue identically!

First, a helpful trick for reproducing this -- add a 1-3 second sleep before pivoting root:
https://github.com/opencontainers/runc/blob/593914b8bd5448a93f7c3e4902a03408b6d5c0ce/libcontainer/rootfs_linux.go#L98-L103

And a few hundred ms in the Put code here:

defer d.locker.Unlock(id)

After those changes, it reliably reproduces running just two containers at once, which made it much easier to continue investigating.

Anyways, it turns out that if you look through every mount namespace for references to the overlayfs mount, you'll find that the runc init process for another container sometimes has a copy of it still mounted, despite it being unmounted and gone from the host mount namespace.

This copy is a private copy of the mount meaning our host umount won't get it, and we're at the mercy of this other container's runc init to eventually clean it up.

I've created a commented reproduction of what docker and runc are doing namespace and mount wise leading up to this ebusy:

#!/bin/bash
set -x

# c1 and c2 represent two different docker containers starting at once
c1=1
c2=2

function ovlOpts() {
	echo -n "lowerdir=$tmpdir/lower,upperdir=$tmpdir/$1/diff,workdir=$tmpdir/$1/work"
}

tmpdir=$(mktemp -d) # 'overlay2' graphdriver dir

mkdir -p $tmpdir/{$c1,$c2}/{diff,merged,work}
mkdir -p $tmpdir/lower

# overlay2 driver in its setup code does this
# https://github.com/moby/moby/blob/ba317637de9b9918cdc2139466dd51c6200bd158/daemon/graphdriver/overlay2/overlay.go#L178
mount --bind $tmpdir $tmpdir
mount --make-private $tmpdir

# Container 2 sets up 
# https://github.com/moby/moby/blob/ba317637de9b9918cdc2139466dd51c6200bd158/daemon/graphdriver/overlay2/overlay.go#L589
mount -t overlay overlay -o "$(ovlOpts $c2)" $tmpdir/$c2/merged 

# Container 1 starts setting up
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged 

# Container 2 runs 'runc init' code in parallel
(
  # https://github.com/opencontainers/runc/blob/8b47a242a9aebdfe1c0c2b6513368f736d505bf0/libcontainer/nsenter/nsexec.c#L823
  unshare -m --propagation unchanged -- bash <<EOF
  # Now runc init remounts /
  # https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605
  # Due to how the config conversion works, config.RootPropagation is never 0,
  # and defaults instead to MS_PRIVATE | MS_REC. I'll PR a fix
  mount --make-rprivate /
  # Now a bunch of init stuff happens, including premount cmds and hooks
  sleep 1
  # .. and then pivot root happens which cleans up our old root
  # It's hard to do in shell, so we'll just pretend an umount of / is close enough
  # https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L676
  cd /
  umount -l .
EOF
) &
   
sleep 0.5

# While container2 is doing its init, container 1 unmounts and remounts its overlay
umount $tmpdir/$c1/merged
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged 
# Boom, EBUSY on 4.13+ because `unshare -m` above has a private copy of the mount

sleep 1
# Now that the runc init code has pivoted and umounted its old root, we're able to mount without EBUSY
mount -t overlay overlay -o "$(ovlOpts $c1)" $tmpdir/$c1/merged 
umount $tmpdir/$c1/merged

# Cleanup
umount $tmpdir/$c2/merged
umount $tmpdir
rm -rf $tmpdir

Changing the runc-init mount to be rslave (as I think it was meant to be) and removing the MakePrivate call for the overlay2 graphdriver directory fixes the race. Even with the addition of the above-mentioned sleeps, I no longer am able to get EBUSY with those changes.

I'll PR each of those changes shortly with suitable commit messages.

euank added a commit to euank/runc that referenced this issue Sep 22, 2017
The code in prepareRoot (https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605)
attempts to default the rootfs mount to `rslave`. However, since the spec
conversion has already defaulted it to `rprivate`, that code doesn't
actually ever do anything.

This changes the spec conversion code to accept "" and treat it as 0.

Implicitly, this makes rootfs propagation default to `rslave`, which is
a part of fixing the moby bug moby/moby#34672

Alternate implementatoins include changing this defaulting to be
`rslave` and removing the defaulting code in prepareRoot, or skipping
the mapping entirely for "", but I think this change is the cleanest of
those options.
@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 22, 2017

Thanks for the extensive analysis @euank

euank added a commit to euank/runc that referenced this issue Sep 22, 2017
The code in prepareRoot (https://github.com/opencontainers/runc/blob/e385f67a0e45fa1d8ef8154e2aea5128ea1d331b/libcontainer/rootfs_linux.go#L599-L605)
attempts to default the rootfs mount to `rslave`. However, since the spec
conversion has already defaulted it to `rprivate`, that code doesn't
actually ever do anything.

This changes the spec conversion code to accept "" and treat it as 0.

Implicitly, this makes rootfs propagation default to `rslave`, which is
a part of fixing the moby bug moby/moby#34672

Alternate implementatoins include changing this defaulting to be
`rslave` and removing the defaulting code in prepareRoot, or skipping
the mapping entirely for "", but I think this change is the cleanest of
those options.

Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
euank added a commit to euank/docker that referenced this issue Sep 22, 2017
This reverts commit e076bcc.
It also reverts it for the overlay2 package, which didn't exist at the
time the commit was made but is a direct successor with copy-pasted
code.

The original commit was *meant* to fix a bug whereby `docker cp`
(implemented via chrootarchive) could inadvertantly lead to shared
mounts getting unmounted on the host too.

The fix, however, is wrong. It results in unmounting overlay mounts
reliably, in the face of multiple mount namespaces, being impossible.

This hasn't been noticed until now because on kernels prior to v4.13,
temporarily leaking overlayfs mounts to other namespaces didn't have any
ill effects.

Starting with v4.13, setting the mount to private and thus leaking
mounts results in errors. See moby#34672

The correct fix for the original issue was implemented later in
moby#27609, and since that code is now
merged we can safely throw away this incorrect fix.

Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
euank added a commit to euank/docker that referenced this issue Sep 22, 2017
This reverts commit e076bcc.
It also reverts it for the overlay2 package, which didn't exist at the
time the commit was made but is a direct successor with copy-pasted
code.

The original commit was *meant* to fix a bug whereby `docker cp`
(implemented via chrootarchive) could inadvertantly lead to shared
mounts getting unmounted on the host too.

The fix, however, is wrong. It results in unmounting overlay mounts
reliably, in the face of multiple mount namespaces, being impossible.

This hasn't been noticed until now because on kernels prior to v4.13,
temporarily leaking overlayfs mounts to other namespaces didn't have any
ill effects.

Starting with v4.13, setting the mount to private and thus leaking
mounts results in errors. See moby#34672

The correct fix for the original issue was implemented later in
moby#27609, and since that code is now
merged we can safely throw away this incorrect fix.

Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
@euank
Copy link
Contributor Author

@euank euank commented Sep 22, 2017

In my testing, this is fixed by applying both #34948 and opencontainers/runc#1598

One or the other isn't enough since both result in a private copy of the mount off in runc's init mountns, but both of them together ensure it doesn't happen.

euank added a commit to euank/docker that referenced this issue Sep 22, 2017
This reverts commit e076bcc.
It also reverts it for the overlay2 package, which didn't exist at the
time the commit was made but is a direct successor with copy-pasted
code.

The original commit was meant to fix a bug whereby `docker cp`
(implemented via chrootarchive) could inadvertantly lead to shared
mounts getting unmounted on the host too.

The fix, however, had side effects. It results in overlay mounts being
private, and thus being quite easy to leak copies that are hard to
umount into other mount namespaces on the box.

This hasn't been noticed until now because on kernels prior to v4.13,
temporarily leaking overlayfs mounts to other namespaces didn't have any
ill effects.

Starting with v4.13, setting the mount to private and thus leaking
mounts results in errors. See moby#34672

The correct fix for the original issue was implemented later in
moby#27609, and since that code is now
merged we can safely throw away this less ideal fix.

Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
@euank
Copy link
Contributor Author

@euank euank commented Sep 22, 2017

I think it's still possible to hit the race in theory without also applying opencontainers/runc#1500, but in practice I can't hit that one.

euank added a commit to euank/coreos-overlay that referenced this issue Sep 22, 2017
See coreos/bugs#2127 and
moby/moby#34672 for discussion.

Patch files have been split into more folders, plus some manual eapply
calls, to allow moby/moby patches to be used unmodified against
docker-ce
@cyphar
Copy link
Contributor

@cyphar cyphar commented Sep 22, 2017

This looks like another variant of #34573 (effectively mount leaks are causing EBUSY in a variety of places). Honestly we should re-architect Docker/containerd so that it spawns runC in different mount namespaces (with the rootfs mount only done privately in each mount namespace) so that the mounts won't be leakable to other runcs.

euank added a commit to euank/coreos-overlay that referenced this issue Sep 22, 2017
See coreos/bugs#2127 and
moby/moby#34672 for discussion.

Patch files have been split into more folders, plus some manual eapply
calls, to allow moby/moby patches to be used unmodified against
docker-ce
euank added a commit to euank/coreos-overlay that referenced this issue Sep 22, 2017
See coreos/bugs#2127 and
moby/moby#34672 for discussion.

Patch files have been split into more folders, plus some manual eapply
calls, to allow moby/moby patches to be used unmodified against
docker-ce
@maltris
Copy link

@maltris maltris commented Sep 27, 2017

Hi, I think I can confirm this behaviour as of kernel 4.13.3 and docker-ce 17.09!

docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:24 2017
 OS/Arch:      linux/amd64
 Experimental: false

dmesg:

overlayfs: upperdir is in-use by another mount

docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 109
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.3-041303-generic
Operating System: Ubuntu 17.04
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.669GiB
Name: hostname
ID: something
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

@1arrow
Copy link

@1arrow 1arrow commented Mar 7, 2018

Will upgrade the kernel version and check

amir73il added a commit to amir73il/linux that referenced this issue May 24, 2018
Enforcing exclusive ownership on upper/work dirs caused a docker
regression: moby/moby#34672.

Euan spotted the regression and pointed to the offending commit.
Vivek has brought the regression to my attention and provided this
reproducer:

Terminal 1:

  mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none
        merged/

Terminal 2:

  unshare -m

Terminal 1:

  umount merged
  mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none
        merged/
  mount: /root/overlay-testing/merged: none already mounted or mount point
         busy

To fix the regression, I replaced the error with an alarming warning.
With index feature enabled, mount does fail, but logs a suggestion to
override exclusive dir protection by disabling index.
Note that index=off mount does take the inuse locks, so a concurrent
index=off will issue the warning and a concurrent index=on mount will fail.

Documentation was updated to reflect this change.

Fixes: 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs")
Cc: <stable@vger.kernel.org> # v4.13
Reported-by: Euan Kemp <euank@euank.com>
Reported-by: Vivek Goyal <vgoyal@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
amir73il added a commit to amir73il/linux that referenced this issue Nov 4, 2018
Enforcing exclusive ownership on upper/work dirs caused a docker
regression: moby/moby#34672.

Euan spotted the regression and pointed to the offending commit.
Vivek has brought the regression to my attention and provided this
reproducer:

Terminal 1:

  mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none
        merged/

Terminal 2:

  unshare -m

Terminal 1:

  umount merged
  mount -t overlay -o workdir=work,lowerdir=lower,upperdir=upper none
        merged/
  mount: /root/overlay-testing/merged: none already mounted or mount point
         busy

To fix the regression, I replaced the error with an alarming warning.
With index feature enabled, mount does fail, but logs a suggestion to
override exclusive dir protection by disabling index.
Note that index=off mount does take the inuse locks, so a concurrent
index=off will issue the warning and a concurrent index=on mount will fail.

Documentation was updated to reflect this change.

Fixes: 2cac0c0 ("ovl: get exclusive ownership on upper/work dirs")
Cc: <stable@vger.kernel.org> # v4.13
Reported-by: Euan Kemp <euank@euank.com>
Reported-by: Vivek Goyal <vgoyal@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
borkmann added a commit to cilium/cilium that referenced this issue Jun 25, 2019
Same process to get in latest bpftool and iproute2 changes.
For net-next, the build 28 went fine, but CI is hitting the
docker bug in moby/moby#34672,
leave it out for now.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
borkmann added a commit to cilium/cilium that referenced this issue Jun 25, 2019
Same process to get in latest bpftool and iproute2 changes.
For net-next, the build 28 went fine, but CI is hitting the
docker bug in moby/moby#34672,
leave it out for now.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
amir73il added a commit to amir73il/linux that referenced this issue Jul 11, 2019
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
amir73il added a commit to amir73il/linux that referenced this issue Jul 11, 2019
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
amir73il added a commit to amir73il/linux that referenced this issue Jul 14, 2019
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
avagin pushed a commit to avagin/linux that referenced this issue Jul 17, 2019
Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
mrchapp pushed a commit to mrchapp/linux that referenced this issue Sep 20, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Whissi pushed a commit to Whissi/linux-stable that referenced this issue Sep 21, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
woodsts pushed a commit to woodsts/linux-stable that referenced this issue Sep 21, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
woodsts pushed a commit to woodsts/linux-stable that referenced this issue Sep 21, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
isjerryxiao added a commit to isjerryxiao/Amlogic_s905-kernel that referenced this issue Sep 21, 2019
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e5a586 ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
mrchapp pushed a commit to mrchapp/linux that referenced this issue Sep 21, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
jpuhlman pushed a commit to MontaVista-OpenSourceTechnology/linux-mvista-2.6 that referenced this issue Sep 24, 2019
Source: Kernel.org
MR: 100044
Type: Integration
Disposition: Backport from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable linux-4.19.y
ChangeID: 9c13e770cca978b8d5c870f2270cdf3a10c63817
Description:

commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e5a586 ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster@mvista.com>
tobetter added a commit to tobetter/linux that referenced this issue Oct 1, 2019
commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
jackpot51 pushed a commit to pop-os/linux that referenced this issue Oct 14, 2019
BugLink: https://bugs.launchpad.net/bugs/1845642

commit 0be0bfd upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
Staars added a commit to Staars/linux-stable that referenced this issue Oct 15, 2019
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e5a586 ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
brauner pushed a commit to brauner/ubuntu-disco that referenced this issue Oct 16, 2019
BugLink: https://bugs.launchpad.net/bugs/1846277

commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e5a586 ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Kamal Mostafa <kamal@canonical.com>
Signed-off-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
casparant pushed a commit to alibaba/cloud-kernel that referenced this issue Oct 30, 2019
commit 0be0bfd2de9dfdd2098a9c5b14bdd8f739c9165d upstream.

Once upon a time, commit 2cac0c0 ("ovl: get exclusive ownership on
upper/work dirs") in v4.13 added some sanity checks on overlayfs layers.
This change caused a docker regression. The root cause was mount leaks
by docker, which as far as I know, still exist.

To mitigate the regression, commit 85fdee1 ("ovl: fix regression
caused by exclusive upper/work dir protection") in v4.14 turned the
mount errors into warnings for the default index=off configuration.

Recently, commit 146d62e5a586 ("ovl: detect overlapping layers") in
v5.2, re-introduced exclusive upper/work dir checks regardless of
index=off configuration.

This changes the status quo and mount leak related bug reports have
started to re-surface. Restore the status quo to fix the regressions.
To clarify, index=off does NOT relax overlapping layers check for this
ovelayfs mount. index=off only relaxes exclusive upper/work dir checks
with another overlayfs mount.

To cover the part of overlapping layers detection that used the
exclusive upper/work dir checks to detect overlap with self upper/work
dir, add a trap also on the work base dir.

Link: moby/moby#34672
Link: https://lore.kernel.org/linux-fsdevel/20171006121405.GA32700@veci.piliscsaba.szeredi.hu/
Link: containers/podman#3540
Fixes: 146d62e5a586 ("ovl: detect overlapping layers")
Cc: <stable@vger.kernel.org> # v4.19+
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Tested-by: Colin Walters <walters@verbum.org>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
Acked-by: Joseph Qi <joseph.qi@linux.alibaba.com>
@AbhinayGupta741
Copy link

@AbhinayGupta741 AbhinayGupta741 commented Nov 13, 2019

$ vim /etc/docker/daemon.json
{
"max-concurrent-uploads": 1
}
$ service docker restart

I had the same issue with the docker. It was not able to push images to GCR. The above will able to solved my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet