Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2.

[darrenscerri]

Description

Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2.

Steps to reproduce the issue:

_Scenario 1_

Try to build the following image

FROM centos:6

RUN \
    rm -rf /var/lib/yum/yumdb && /
    echo END

_Scenario 2_

1. docker run -it centos
2. rm /bin/egrep
3. ls -lhart | grep egrep

Describe the results you received:

_Scenario 1_

Step 1 : FROM centos:latest
 ---> 980e0e4c79ec
Step 2 : RUN rm -rf /var/lib/yum/yumdb && /
 ---> Running in 6d563251bb19
rm: cannot remove '/var/lib/yum/yumdb/a/85e19f5abe37c08a77c42f24624d852a338583bd-acl-2.2.51-12.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/a/b79b02a804fda6e4a10d4c1a68975a323363c2b2-audit-libs-2.4.1-5.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/4e3611aa330e1eb8c42401709d03c6b3746b8a75-binutils-2.23.52.0.1-55.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/62be349c36ee42d628776a98d2d4b688a9fe4203-bash-4.2.46-20.el7_2-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/cdf13cd90ccdecfc009b4cb319471faa995a9fa3-bind-license-9.9.4-29.el7_2.3-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/ed5c44b6d81784129138935e009a5d71fcca2e22-basesystem-10.0-7.el7.centos-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/fdf683c5a9852bb4ccf883ff042042923b8cd33f-bzip2-libs-1.0.6-13.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/0ec1f87be823c6009e3635d19d9c4302e19d4575-cpio-2.11-24.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/57bf8951d232f117c7e19bbcebfc6a40d15f7e78-cryptsetup-libs-1.6.7-1.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/5a28ca24d3ccf264d0009576e41df09a4400b499-cracklib-dicts-2.9.0-11.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/6e1e6faa0142bab65d7342c0b67b838c9abe07ef-ca-certificates-2015.2.6-70.1.el7_2-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/75c7de956ef79c6d91a2049523fc4fec931e8990-coreutils-8.22-15.el7_2.1-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/7c6a1ce1ee0ff382b98e0f584ad5a5b3ba523785-chkconfig-1.3.61-5.el7_2.1-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/988d2338f7dc11e0c5b63c2165b488b80cec883b-centos-release-7-2.1511.el7.centos.2.10-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/bc5c88cecec32eb303b45fc24611a2d859603bc9-cyrus-sasl-lib-2.1.26-20.el7_2-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/c1a9cb40fa6041bdfb09fb82c63aa8c6b34b3345-cracklib-2.9.0-11.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/c1fefbbaf7e3b7e7c756f71d6856faa8c0fae569-curl-7.29.0-25.el7.centos-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/d/067d41965a62020e76eea1686386e17cc178ac63-device-mapper-libs-1.02.107-5.el7_2.5-x86_64': Directory not empty

_Scenario 2_

[root@04f727c9abcc bin]# ls -lhart | grep egrep
ls: cannot access egrep: No such file or directory
??????????? ? ?    ?       ?            ? egrep
-rwxr-xr-x. 1 root root  123 Jul 27  2015 zegrep
lrwxrwxrwx. 1 root root    6 Sep  6 14:00 xzegrep -> xzgrep

Describe the results you expected:

Files should be removed correctly without errors.

Output of docker version:

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 21
 Running: 13
 Paused: 0
 Stopped: 8
Images: 19
Server Version: 1.12.2
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.8.1-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.848 GiB
Name: host2
ID: AEWS:D74L:S5HN:F6TX:KSYL:FMBO:KF4C:AMME:CCT6:THJB:2PRH:VW7T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details:

VM on Hyper-V

It looks to be a kernel Issue as it is also mentioned here: https://bugs.centos.org/view.php?id=11878&nbn=1. Posted here for visibility

[thaJeztah]

Looks like you're running a custom kernel (4.8), instead of the 3.10 kernel provided by CentOS; note that we generally don't support that, especially on CentOS/RHEL, as there's a lot of back ports done by Red Hat on those distro's, which can be very specific.

Thanks for adding that link though, I'll add "overlay" and "kernel" labels

/cc @dmcgowan @AkihiroSuda

[AkihiroSuda]

Couldn't hit in my environment

• Ubuntu 16.04
• Kernel 4.8.1 vanilla (config is from Ubuntu 16.04's kernel 4.4.0)
• Docker 1.12.2
• Storage driver is overlay2 (tested both XFS and ext4 as a backing filesystem)

$ docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.12.2
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.8.1-ldlk+
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.613 GiB
Name: tmp01
ID: JMBJ:3DS5:TXXR:H3NW:W2VB:36VO:SCFM:LXZM:L4YD:VB2R:BUSE:HCXZ
Docker Root Dir: /mnt
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

So the issue seems specific to ELRepo's patches/configs?

I'll try to set up a CentOS7 + ELRepo kernel env later..
(update: yes, I confirmed it is reproducible on CentOS7 + ELRepo kernel 4.8.1-1)

[AkihiroSuda]

On CentOS7 + ELRepo kernel 4.8.1-1, I can reproduce the issue with xfs but not with ext4

[darrenscerri]

@thaJeztah I managed to replicate also using overlay using 3.18 kernel.

After further investigation, I noticed that my xfs partition had ftype set to 0. Red Hat states that xfs partitions need to be formatted with ftype set to 1 if they are to be used with overlay.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

I guess the issue can now be closed as it's not Docker related. Hopefully people who encounter this issue can stumble upon this issue for a solution.

[thaJeztah]

@darrenscerri oh, interesting; would this be something we should check for in our check-config script? https://github.com/docker/docker/blob/master/contrib/check-config.sh (Or from docker itself even when using the overlay driver) not sure if it's easy to add a check for this

[AkihiroSuda]

rkt seems detecting this: https://github.com/coreos/rkt/pull/3105/files#diff-b5ccff3f3777d1b87c84277733a8c04fR393

I'll try to make a PR in similar way

Also, recent kernel shows a warning (not an error) in dmesg about this issue torvalds/linux@e7c0b59