New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overlayfs: Can't delete file moved from base layer to newly created dir even on ext4 #9572

Closed
overplumbum opened this Issue Dec 8, 2014 · 27 comments

Comments

Projects
None yet
@overplumbum

overplumbum commented Dec 8, 2014

I'm testing overlayfs with docker from master branch 58ce014 ( https://master.dockerproject.com/ ):

Server version: 1.3.2-dev
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 58ce014

On Ubuntu:14.04 with 3.18 kernel from vivid ( http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-vivid/ )

Linux 3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# docker -D info
Containers: 51
Images: 169
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.0-031800-generic
Operating System: Ubuntu 14.04.1 LTS
Debug mode (server): false
Debug mode (client): true
Fds: 10
Goroutines: 13
EventsListeners: 0
Init Path: /usr/local/bin/docker-1.3.2-dev
WARNING: No swap limit support

With graph stored on EXT4 partition using overlay(fs) storage backend.

To reproduce issue one should:

  • create new dir
  • move some file from base layer to this new dir
  • remove this file from it's new location (no error message here)
  • check if it is still there (file transforms to some kind of character device)

Snippet to reproduce:

$ docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'
+ mkdir /foo
+ mv /bin/false /foo
+ rm -v /foo/false
+ ls -l /foo
total 0
c---------    1 root     root        0,   0 Dec  8 22:51 false
+ rmdir /foo
rmdir: '/foo': Directory not empty

@overplumbum overplumbum changed the title from overlayfs: Can't delete file moved from upper layer to newly created dir even on ext4 to overlayfs: Can't delete file moved from base layer to newly created dir even on ext4 Dec 8, 2014

@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Dec 9, 2014

Contributor

Trying this now

On Monday, December 8, 2014, Denis Orlikhin notifications@github.com
wrote:

I'm testing overlayfs with docker from master branch 58ce014
58ce014
( https://master.dockerproject.com/ ):

Server version: 1.3.2-dev
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 58ce014

On Ubuntu:14.04 with 3.18 kernel from vivid (
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-vivid/ )

Linux 3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

docker -D info

Containers: 51
Images: 169
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.0-031800-generic
Operating System: Ubuntu 14.04.1 LTS
Debug mode (server): false
Debug mode (client): true
Fds: 10
Goroutines: 13
EventsListeners: 0
Init Path: /usr/local/bin/docker-1.3.2-dev
WARNING: No swap limit support

With graph stored on EXT4 partition using overlay(fs) storage backend.

To reproduce issue one should:

  • create new dir
  • move some file from upper layer to this new dir
  • remove this file from it's new location (no error message here)
  • check if it is still there (file transforms to some kind of
    character device)

Snippet to reproduce:

$ docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'

  • mkdir /foo
  • mv /bin/false /foo
  • rm -v /foo/false
  • ls -l /foo
    total 0
    c--------- 1 root root 0, 0 Dec 8 22:51 false
  • rmdir /foo
    rmdir: '/foo': Directory not empty


Reply to this email directly or view it on GitHub
#9572.

Contributor

jessfraz commented Dec 9, 2014

Trying this now

On Monday, December 8, 2014, Denis Orlikhin notifications@github.com
wrote:

I'm testing overlayfs with docker from master branch 58ce014
58ce014
( https://master.dockerproject.com/ ):

Server version: 1.3.2-dev
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 58ce014

On Ubuntu:14.04 with 3.18 kernel from vivid (
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-vivid/ )

Linux 3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

docker -D info

Containers: 51
Images: 169
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.0-031800-generic
Operating System: Ubuntu 14.04.1 LTS
Debug mode (server): false
Debug mode (client): true
Fds: 10
Goroutines: 13
EventsListeners: 0
Init Path: /usr/local/bin/docker-1.3.2-dev
WARNING: No swap limit support

With graph stored on EXT4 partition using overlay(fs) storage backend.

To reproduce issue one should:

  • create new dir
  • move some file from upper layer to this new dir
  • remove this file from it's new location (no error message here)
  • check if it is still there (file transforms to some kind of
    character device)

Snippet to reproduce:

$ docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'

  • mkdir /foo
  • mv /bin/false /foo
  • rm -v /foo/false
  • ls -l /foo
    total 0
    c--------- 1 root root 0, 0 Dec 8 22:51 false
  • rmdir /foo
    rmdir: '/foo': Directory not empty


Reply to this email directly or view it on GitHub
#9572.

@snnn

This comment has been minimized.

Show comment
Hide comment
@snnn

snnn Feb 5, 2015

I can reproduce this bug on Fedora 21.
Docker version: docker-io-1.4.1-8.fc21.x86_64
Kernel version: Linux xxxxx 3.18.3-201.fc21.x86_64 #1 SMP Mon Jan 19 15:59:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

snnn commented Feb 5, 2015

I can reproduce this bug on Fedora 21.
Docker version: docker-io-1.4.1-8.fc21.x86_64
Kernel version: Linux xxxxx 3.18.3-201.fc21.x86_64 #1 SMP Mon Jan 19 15:59:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

@ctapan

This comment has been minimized.

Show comment
Hide comment
@ctapan

ctapan Mar 24, 2015

I am not able to reproduce this issue on Ubuntu with older "overlayfs" driver and docker 1.4.1.
3.13.0-44-generic 73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Can someone paste the output for following?
ls -lah /var/lib/docker/overlay//upper/
ls -lah /var/lib/docker/overlay//upper/foo/
ls -lah /var/lib/docker/overlay//upper/bin/

(Maybe with latest "overlay" driver, last directory might be in /work/)

Just trying to understand if this is overlay issue or chroot. Most likely will be overlay driver. Sorry I do not have the latest kernel version to try.

ctapan commented Mar 24, 2015

I am not able to reproduce this issue on Ubuntu with older "overlayfs" driver and docker 1.4.1.
3.13.0-44-generic 73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Can someone paste the output for following?
ls -lah /var/lib/docker/overlay//upper/
ls -lah /var/lib/docker/overlay//upper/foo/
ls -lah /var/lib/docker/overlay//upper/bin/

(Maybe with latest "overlay" driver, last directory might be in /work/)

Just trying to understand if this is overlay issue or chroot. Most likely will be overlay driver. Sorry I do not have the latest kernel version to try.

@sgirones

This comment has been minimized.

Show comment
Hide comment
@sgirones

sgirones Apr 14, 2015

Reproduced:

$  docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'
+ mkdir /foo
+ mv /bin/false /foo
+ rm -v /foo/false
+ ls -l /foo
total 0
c---------    1 root     root        0,   0 Apr 14 11:00 false
+ rmdir /foo
rmdir: '/foo': Directory not empty

$ uname -a
Linux product-builder-1 3.18.11-031811-generic #201504041535 SMP Sat Apr 4 19:37:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Containers: 4
Images: 136
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.18.11-031811-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 23.95 GiB
Name: product-builder-1
ID: GCJZ:YZXU:2V74:NNEE:UUXB:BOC6:LNNC:EELA:C2P6:4DEO:3SJE:N2NZ
WARNING: No swap limit support

sgirones commented Apr 14, 2015

Reproduced:

$  docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'
+ mkdir /foo
+ mv /bin/false /foo
+ rm -v /foo/false
+ ls -l /foo
total 0
c---------    1 root     root        0,   0 Apr 14 11:00 false
+ rmdir /foo
rmdir: '/foo': Directory not empty

$ uname -a
Linux product-builder-1 3.18.11-031811-generic #201504041535 SMP Sat Apr 4 19:37:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Containers: 4
Images: 136
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.18.11-031811-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 23.95 GiB
Name: product-builder-1
ID: GCJZ:YZXU:2V74:NNEE:UUXB:BOC6:LNNC:EELA:C2P6:4DEO:3SJE:N2NZ
WARNING: No swap limit support
@sgirones

This comment has been minimized.

Show comment
Hide comment
@sgirones

sgirones Apr 14, 2015

Reproduced even with 4.0.0. But behavior was slightly different:

$ uname -a
Linux product-builder-1 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'
+ mkdir /foo
+ mv /bin/false /foo
+ rm -v /foo/false
+ ls -l /foo
ls: /foo/false: No such file or directory
total 0

sgirones commented Apr 14, 2015

Reproduced even with 4.0.0. But behavior was slightly different:

$ uname -a
Linux product-builder-1 4.0.0-040000-generic #201504121935 SMP Sun Apr 12 23:36:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ docker run busybox sh -xec 'mkdir /foo ; mv /bin/false /foo ; rm -v /foo/false ; ls -l /foo ; rmdir /foo'
+ mkdir /foo
+ mv /bin/false /foo
+ rm -v /foo/false
+ ls -l /foo
ls: /foo/false: No such file or directory
total 0
@sgirones

This comment has been minimized.

Show comment
Hide comment
@davebirch

This comment has been minimized.

Show comment
Hide comment
@davebirch

davebirch May 26, 2015

As an additional piece of potentially useful info, I'm seeing this same issue when using pure lxc with an overlayfs filesystem.

If I delete a file from the lower filesystem, the file is not removed but becomes highlighted as if it's a broken symlink. Trying to read the file results in a "No such file or directory" error.

Current versions:
Linux testcontainer 3.13.0-52-generic #85-Ubuntu SMP Wed Apr 29 16:44:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

LXC versions 1.0.7 & 1.1.2

After doing some testing on a couple of different machines, I discovered that this problem occurs on one box I have where the host filesystem is formatted to xfs, but does not occur on my local machine here the / filesystem is formatted to ext4

davebirch commented May 26, 2015

As an additional piece of potentially useful info, I'm seeing this same issue when using pure lxc with an overlayfs filesystem.

If I delete a file from the lower filesystem, the file is not removed but becomes highlighted as if it's a broken symlink. Trying to read the file results in a "No such file or directory" error.

Current versions:
Linux testcontainer 3.13.0-52-generic #85-Ubuntu SMP Wed Apr 29 16:44:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

LXC versions 1.0.7 & 1.1.2

After doing some testing on a couple of different machines, I discovered that this problem occurs on one box I have where the host filesystem is formatted to xfs, but does not occur on my local machine here the / filesystem is formatted to ext4

@darlingm

This comment has been minimized.

Show comment
Hide comment
@darlingm

darlingm Jun 8, 2015

EDIT: FWIW, on Fedora 22 (kernel 4.0.4-303.fc22.x86_64), I CAN reproduce this bug using overlay and chroot. (I'm not running lxc or docker.) ls -la /foo shows:

ls: cannot access foo/false: No such file or directory
total 0
c????????? ? ? ? ?            ? false

And rmdir foo shows: "rmdir: failed to remove 'foo': Directory not empty"

darlingm commented Jun 8, 2015

EDIT: FWIW, on Fedora 22 (kernel 4.0.4-303.fc22.x86_64), I CAN reproduce this bug using overlay and chroot. (I'm not running lxc or docker.) ls -la /foo shows:

ls: cannot access foo/false: No such file or directory
total 0
c????????? ? ? ? ?            ? false

And rmdir foo shows: "rmdir: failed to remove 'foo': Directory not empty"

@vikstrous

This comment has been minimized.

Show comment
Hide comment
@vikstrous

vikstrous Nov 19, 2015

Contributor

This is easy to reproduce with nothing any more special than mount.

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
/usr/bin/ls: cannot access merged/dir/test: No such file or directory
total 0
c????????? ? ? ? ?            ? test
$ uname -a
Linux linusputinmusk 4.2.2-1-ARCH #1 SMP PREEMPT Tue Sep 29 22:21:33 CEST 2015 x86_64 GNU/Linux

Has this bug been reported to the linux kernel / whoever is working on overlay?

Contributor

vikstrous commented Nov 19, 2015

This is easy to reproduce with nothing any more special than mount.

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
/usr/bin/ls: cannot access merged/dir/test: No such file or directory
total 0
c????????? ? ? ? ?            ? test
$ uname -a
Linux linusputinmusk 4.2.2-1-ARCH #1 SMP PREEMPT Tue Sep 29 22:21:33 CEST 2015 x86_64 GNU/Linux

Has this bug been reported to the linux kernel / whoever is working on overlay?

@vikstrous

This comment has been minimized.

Show comment
Hide comment
@vikstrous

vikstrous Nov 19, 2015

Contributor

Oops, I didn't see the link above. Sorry.

It looks like their repro doesn't move the file to a different directory. I don't think this is the same bug. Another hint is that it was marked as fixed and their repro succeeds while ours fails.

So far it looks like it happens only when the directory where the file is moved to and deleted from doesn't exist in the lower layer.

Seems like the issue is that it's creating a whiteout file for a file that never existed in the lower layer. It puts the whiteout file at the last location where the original file was moved to. If that dir doesn't exist in the lower tree the whiteout file shows up in the merged tree as a broken character device file.

This is very clearly a pretty serious bug in overlay. I'm surprised that it was not a bigger problem yet.

Edit: one more thing: moving the file a second time reproduces the issue as well - it's equivalent to deleting it

Edit 2: This case succeeds:

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ cp lower/test upper/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
total 0

The difference is that the file was copied up BEFORE the overlay mount was created. If it's copied up after, the bug shows up.

Edit 3: I confirmed that unmounting and re-mounting after coping the file up fixes the issue:

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ touch upper/test
$ sudo umount merged
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
total 0

There is some sort of state associated with this file after it's copied up that affects the behavior when removing it. This state doesn't persist after re-mounting.

Contributor

vikstrous commented Nov 19, 2015

Oops, I didn't see the link above. Sorry.

It looks like their repro doesn't move the file to a different directory. I don't think this is the same bug. Another hint is that it was marked as fixed and their repro succeeds while ours fails.

So far it looks like it happens only when the directory where the file is moved to and deleted from doesn't exist in the lower layer.

Seems like the issue is that it's creating a whiteout file for a file that never existed in the lower layer. It puts the whiteout file at the last location where the original file was moved to. If that dir doesn't exist in the lower tree the whiteout file shows up in the merged tree as a broken character device file.

This is very clearly a pretty serious bug in overlay. I'm surprised that it was not a bigger problem yet.

Edit: one more thing: moving the file a second time reproduces the issue as well - it's equivalent to deleting it

Edit 2: This case succeeds:

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ cp lower/test upper/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
total 0

The difference is that the file was copied up BEFORE the overlay mount was created. If it's copied up after, the bug shows up.

Edit 3: I confirmed that unmounting and re-mounting after coping the file up fixes the issue:

$ mkdir upper lower work merged upper/dir/
$ touch lower/test
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ touch upper/test
$ sudo umount merged
$ sudo mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged
$ mv merged/test merged/dir/
$ rm merged/dir/test
$ ls -l merged/dir/
total 0

There is some sort of state associated with this file after it's copied up that affects the behavior when removing it. This state doesn't persist after re-mounting.

@overplumbum

This comment has been minimized.

Show comment
Hide comment
@overplumbum

overplumbum Nov 19, 2015

I've some time debugging this issue in kernel.

I can confirm your idea about state caching — remounting helps because of dentries cache flush. And overlay driver stores metadata exactly in dentry nodes. Looks like VFS swaps nodes with it's meta during rename: thing overlay neither expect nor able to control.

overplumbum commented Nov 19, 2015

I've some time debugging this issue in kernel.

I can confirm your idea about state caching — remounting helps because of dentries cache flush. And overlay driver stores metadata exactly in dentry nodes. Looks like VFS swaps nodes with it's meta during rename: thing overlay neither expect nor able to control.

@xeor

This comment has been minimized.

Show comment
Hide comment
@xeor

xeor Nov 23, 2015

Example of weirdness:

root@2fefd9a8fee7:/home/git/gitlab# ls -l /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
-rw-r--r--. 11 git git 0 Nov 22 07:23 /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
root@2fefd9a8fee7:/home/git/gitlab# rm -f /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
root@2fefd9a8fee7:/home/git/gitlab# ls -l /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
ls: cannot access /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep: No such file or directory
root@2fefd9a8fee7:/home/git/gitlab# ls -la /home/git/gitlab/shared/artifacts/tmp/cache/
ls: cannot access /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep: No such file or directory
total 0
drwxr-xr-x. 1 git git 21 Nov 23 21:56 .
drwxr-xr-x. 1 git git 18 Nov 23 21:20 ..
??????????? ? ?   ?    ?            ? .gitkeep
root@2fefd9a8fee7:/home/git/gitlab# rm -rf /home/git/gitlab/shared/artifacts/tmp/cache
rm: cannot remove '/home/git/gitlab/shared/artifacts/tmp/cache': Directory not empty

xeor commented Nov 23, 2015

Example of weirdness:

root@2fefd9a8fee7:/home/git/gitlab# ls -l /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
-rw-r--r--. 11 git git 0 Nov 22 07:23 /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
root@2fefd9a8fee7:/home/git/gitlab# rm -f /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
root@2fefd9a8fee7:/home/git/gitlab# ls -l /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep
ls: cannot access /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep: No such file or directory
root@2fefd9a8fee7:/home/git/gitlab# ls -la /home/git/gitlab/shared/artifacts/tmp/cache/
ls: cannot access /home/git/gitlab/shared/artifacts/tmp/cache/.gitkeep: No such file or directory
total 0
drwxr-xr-x. 1 git git 21 Nov 23 21:56 .
drwxr-xr-x. 1 git git 18 Nov 23 21:20 ..
??????????? ? ?   ?    ?            ? .gitkeep
root@2fefd9a8fee7:/home/git/gitlab# rm -rf /home/git/gitlab/shared/artifacts/tmp/cache
rm: cannot remove '/home/git/gitlab/shared/artifacts/tmp/cache': Directory not empty
@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Nov 23, 2015

Contributor

@vikstrous Your case works for me on 4.3.0
However original report now fails for me on ls :

+ ls /foo
ls: /foo/false: No such file or directory

But I can't reproduce it outside docker :/
EDIT: reproduced, I swapped layers occasionally :(

Contributor

LK4D4 commented Nov 23, 2015

@vikstrous Your case works for me on 4.3.0
However original report now fails for me on ls :

+ ls /foo
ls: /foo/false: No such file or directory

But I can't reproduce it outside docker :/
EDIT: reproduced, I swapped layers occasionally :(

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Nov 23, 2015

Contributor

I'm pretty sure it should be easy to fix.

Contributor

LK4D4 commented Nov 23, 2015

I'm pretty sure it should be easy to fix.

@xeor

This comment has been minimized.

Show comment
Hide comment
@xeor

xeor Nov 25, 2015

This problem solved itself for me (as well as for other people in the comments), when going from xfs as a storage backend for overlay to ext4.

xeor commented Nov 25, 2015

This problem solved itself for me (as well as for other people in the comments), when going from xfs as a storage backend for overlay to ext4.

@vikstrous

This comment has been minimized.

Show comment
Hide comment
@vikstrous

vikstrous Dec 19, 2015

Contributor

I opened an issue on the linux kernel bug tracker: https://bugzilla.kernel.org/show_bug.cgi?id=109611

Contributor

vikstrous commented Dec 19, 2015

I opened an issue on the linux kernel bug tracker: https://bugzilla.kernel.org/show_bug.cgi?id=109611

@mishak87

This comment has been minimized.

Show comment
Hide comment
@mishak87

mishak87 Jan 7, 2016

Contributor

I have same issue on Linux 1568a0a4bee9 4.2.0-23-generic #28-Ubuntu using ext4.

Contributor

mishak87 commented Jan 7, 2016

I have same issue on Linux 1568a0a4bee9 4.2.0-23-generic #28-Ubuntu using ext4.

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal
Contributor

rhvgoyal commented Jan 11, 2016

FragLegs added a commit to FragLegs/pip that referenced this issue Jan 27, 2016

In Docker containers with the overlayfs backend, when a
    package is installed at one layer, and then upgraded or downgraded at
    another, such that pip uninstalls the previously installed version, an
    error can occur when files are copied to /tmp (so that they can be rolled
    back) and then deleted on successful uninstall. See discussion about this
    issue at moby/moby#9572 and
    moby/moby#12327. While we are waiting for
    the overlayfs folks to fix this issue, we need to handle the OSError that
    happens when we try to delete a file that "isn't there" according to the
    overlay backend.
@b-a-t

This comment has been minimized.

Show comment
Hide comment
@b-a-t

b-a-t Feb 2, 2016

I've experienced the same problem on Debian 4.3.3-7~bpo8+1 (2016-01-19) kernel with xfs storage backstore for overlay FS. Re-formatting it to ext4 with extra-high i-nodes number seems, solved the problem.

b-a-t commented Feb 2, 2016

I've experienced the same problem on Debian 4.3.3-7~bpo8+1 (2016-01-19) kernel with xfs storage backstore for overlay FS. Re-formatting it to ext4 with extra-high i-nodes number seems, solved the problem.

@koct9i

This comment has been minimized.

Show comment
Hide comment
@koct9i

koct9i Mar 14, 2016

Fixed in linux 4.5 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45d11738969633ec07ca35d75d486bf2d8918df6 going to be backported into next 4.4.y and other stable brances. Simple test sequence in commit message.

koct9i commented Mar 14, 2016

Fixed in linux 4.5 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45d11738969633ec07ca35d75d486bf2d8918df6 going to be backported into next 4.4.y and other stable brances. Simple test sequence in commit message.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 14, 2016

Member

Thanks for the heads up @koct9i

Member

thaJeztah commented Mar 14, 2016

Thanks for the heads up @koct9i

@mountkin

This comment has been minimized.

Show comment
Hide comment
@mountkin

mountkin May 9, 2016

Contributor

I came across a similar but different problem with overlay+xfs on kernel 4.5.3
After deleting a file in the merged directory, ls the file in the merged directory showing all the file attributes as question mark. The file in the upper directory turn out to be a character device.

3

Contributor

mountkin commented May 9, 2016

I came across a similar but different problem with overlay+xfs on kernel 4.5.3
After deleting a file in the merged directory, ls the file in the merged directory showing all the file attributes as question mark. The file in the upper directory turn out to be a character device.

3

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom May 9, 2016

Member

@mountkin http://article.gmane.org/gmane.linux.file-systems.union/496 there's an attempt to fix that but didn't receive any comments yet

Member

runcom commented May 9, 2016

@mountkin http://article.gmane.org/gmane.linux.file-systems.union/496 there's an attempt to fix that but didn't receive any comments yet

@koct9i

This comment has been minimized.

Show comment
Hide comment
@koct9i

koct9i May 9, 2016

@runcom Nope, this is different bug. That you're linked was fixed in 4.5 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45d11738969633ec07ca35d75d486bf2d8918df6 see my comments above

@mountkin this is completely wrong place for reporting linux kernel bugs
Please submit this into https://bugzilla.kernel.org and/or lkml and related mailing list.

I cannot reproduce this right now in 4.4.8+ext4, I'll try exact setup later.

koct9i commented May 9, 2016

@runcom Nope, this is different bug. That you're linked was fixed in 4.5 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45d11738969633ec07ca35d75d486bf2d8918df6 see my comments above

@mountkin this is completely wrong place for reporting linux kernel bugs
Please submit this into https://bugzilla.kernel.org and/or lkml and related mailing list.

I cannot reproduce this right now in 4.4.8+ext4, I'll try exact setup later.

@mountkin

This comment has been minimized.

Show comment
Hide comment
@mountkin

mountkin May 9, 2016

Contributor

Thanks a lot @runcom
The difference between my report and http://article.gmane.org/gmane.linux.file-systems.union/496 is that I don't need to move the file to a sub directory.

@koct9i I'm testing with xfs, not ext4. Let me make some noise on the kernel bugzilla then 😇

Edit: After searching the kernel bugzilla I got this one https://bugzilla.kernel.org/show_bug.cgi?id=108811 , which is exactly the problem I encountered.

Contributor

mountkin commented May 9, 2016

Thanks a lot @runcom
The difference between my report and http://article.gmane.org/gmane.linux.file-systems.union/496 is that I don't need to move the file to a sub directory.

@koct9i I'm testing with xfs, not ext4. Let me make some noise on the kernel bugzilla then 😇

Edit: After searching the kernel bugzilla I got this one https://bugzilla.kernel.org/show_bug.cgi?id=108811 , which is exactly the problem I encountered.

@koct9i

This comment has been minimized.

Show comment
Hide comment
@koct9i

koct9i May 10, 2016

@mountkin (I haven't found your ticket in bugzilla so I leave it here)
linux 4.6 fails to mount overlayfs if upperdir points to xfs created without ftype=1 and prints this "overlayfs: upper fs needs to support d_type.". I guess your xfs was created without this feature.

related commit https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45aebeaf4f67468f76bedf62923a576a519a9b68

koct9i commented May 10, 2016

@mountkin (I haven't found your ticket in bugzilla so I leave it here)
linux 4.6 fails to mount overlayfs if upperdir points to xfs created without ftype=1 and prints this "overlayfs: upper fs needs to support d_type.". I guess your xfs was created without this feature.

related commit https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45aebeaf4f67468f76bedf62923a576a519a9b68

@dmcgowan

This comment has been minimized.

Show comment
Hide comment
@dmcgowan

dmcgowan May 10, 2016

Member

@koct9i thanks for pointing this out. The fix has also been release in 4.4.6 (changelog https://www.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.4.6). Going to close this issue now.

Member

dmcgowan commented May 10, 2016

@koct9i thanks for pointing this out. The fix has also been release in 4.4.6 (changelog https://www.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.4.6). Going to close this issue now.

@dmcgowan dmcgowan closed this May 10, 2016

ChaosEngine added a commit to ChaosEngine/gentoo-docker-images that referenced this issue Sep 19, 2016

dsummersl added a commit to myhpom/MyHPOM that referenced this issue Mar 16, 2018

[#81] Updates docker install instructions.
The version of docker that is in the centos repository is very old and
has a bug that affects the hsctl deploy process (broke the collectstatic
step). The original bug I found that is related to this:

moby/moby#9572
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment