New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2. #27358

Closed
darrenscerri opened this Issue Oct 13, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@darrenscerri

darrenscerri commented Oct 13, 2016

Description

Centos 7 fails to remove files or directories which are created on build of the image using overlay or overlay2.

Steps to reproduce the issue:

_Scenario 1_

Try to build the following image

FROM centos:6

RUN \
    rm -rf /var/lib/yum/yumdb && /
    echo END

_Scenario 2_

  1. docker run -it centos
  2. rm /bin/egrep
  3. ls -lhart | grep egrep

Describe the results you received:

_Scenario 1_

Step 1 : FROM centos:latest
 ---> 980e0e4c79ec
Step 2 : RUN rm -rf /var/lib/yum/yumdb && /
 ---> Running in 6d563251bb19
rm: cannot remove '/var/lib/yum/yumdb/a/85e19f5abe37c08a77c42f24624d852a338583bd-acl-2.2.51-12.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/a/b79b02a804fda6e4a10d4c1a68975a323363c2b2-audit-libs-2.4.1-5.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/4e3611aa330e1eb8c42401709d03c6b3746b8a75-binutils-2.23.52.0.1-55.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/62be349c36ee42d628776a98d2d4b688a9fe4203-bash-4.2.46-20.el7_2-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/cdf13cd90ccdecfc009b4cb319471faa995a9fa3-bind-license-9.9.4-29.el7_2.3-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/ed5c44b6d81784129138935e009a5d71fcca2e22-basesystem-10.0-7.el7.centos-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/b/fdf683c5a9852bb4ccf883ff042042923b8cd33f-bzip2-libs-1.0.6-13.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/0ec1f87be823c6009e3635d19d9c4302e19d4575-cpio-2.11-24.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/57bf8951d232f117c7e19bbcebfc6a40d15f7e78-cryptsetup-libs-1.6.7-1.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/5a28ca24d3ccf264d0009576e41df09a4400b499-cracklib-dicts-2.9.0-11.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/6e1e6faa0142bab65d7342c0b67b838c9abe07ef-ca-certificates-2015.2.6-70.1.el7_2-noarch': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/75c7de956ef79c6d91a2049523fc4fec931e8990-coreutils-8.22-15.el7_2.1-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/7c6a1ce1ee0ff382b98e0f584ad5a5b3ba523785-chkconfig-1.3.61-5.el7_2.1-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/988d2338f7dc11e0c5b63c2165b488b80cec883b-centos-release-7-2.1511.el7.centos.2.10-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/bc5c88cecec32eb303b45fc24611a2d859603bc9-cyrus-sasl-lib-2.1.26-20.el7_2-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/c1a9cb40fa6041bdfb09fb82c63aa8c6b34b3345-cracklib-2.9.0-11.el7-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/c/c1fefbbaf7e3b7e7c756f71d6856faa8c0fae569-curl-7.29.0-25.el7.centos-x86_64': Directory not empty
rm: cannot remove '/var/lib/yum/yumdb/d/067d41965a62020e76eea1686386e17cc178ac63-device-mapper-libs-1.02.107-5.el7_2.5-x86_64': Directory not empty

_Scenario 2_

[root@04f727c9abcc bin]# ls -lhart | grep egrep
ls: cannot access egrep: No such file or directory
??????????? ? ?    ?       ?            ? egrep
-rwxr-xr-x. 1 root root  123 Jul 27  2015 zegrep
lrwxrwxrwx. 1 root root    6 Sep  6 14:00 xzegrep -> xzgrep

Describe the results you expected:

Files should be removed correctly without errors.

Output of docker version:

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 21
 Running: 13
 Paused: 0
 Stopped: 8
Images: 19
Server Version: 1.12.2
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.8.1-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.848 GiB
Name: host2
ID: AEWS:D74L:S5HN:F6TX:KSYL:FMBO:KF4C:AMME:CCT6:THJB:2PRH:VW7T
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details:

VM on Hyper-V

It looks to be a kernel Issue as it is also mentioned here: https://bugs.centos.org/view.php?id=11878&nbn=1. Posted here for visibility

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 15, 2016

Member

Looks like you're running a custom kernel (4.8), instead of the 3.10 kernel provided by CentOS; note that we generally don't support that, especially on CentOS/RHEL, as there's a lot of back ports done by Red Hat on those distro's, which can be very specific.

Thanks for adding that link though, I'll add "overlay" and "kernel" labels

/cc @dmcgowan @AkihiroSuda

Member

thaJeztah commented Oct 15, 2016

Looks like you're running a custom kernel (4.8), instead of the 3.10 kernel provided by CentOS; note that we generally don't support that, especially on CentOS/RHEL, as there's a lot of back ports done by Red Hat on those distro's, which can be very specific.

Thanks for adding that link though, I'll add "overlay" and "kernel" labels

/cc @dmcgowan @AkihiroSuda

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Oct 16, 2016

Member

Couldn't hit in my environment

  • Ubuntu 16.04
  • Kernel 4.8.1 vanilla (config is from Ubuntu 16.04's kernel 4.4.0)
  • Docker 1.12.2
  • Storage driver is overlay2 (tested both XFS and ext4 as a backing filesystem)
$ docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.12.2
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.8.1-ldlk+
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.613 GiB
Name: tmp01
ID: JMBJ:3DS5:TXXR:H3NW:W2VB:36VO:SCFM:LXZM:L4YD:VB2R:BUSE:HCXZ
Docker Root Dir: /mnt
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

So the issue seems specific to ELRepo's patches/configs?

I'll try to set up a CentOS7 + ELRepo kernel env later..
(update: yes, I confirmed it is reproducible on CentOS7 + ELRepo kernel 4.8.1-1)

Member

AkihiroSuda commented Oct 16, 2016

Couldn't hit in my environment

  • Ubuntu 16.04
  • Kernel 4.8.1 vanilla (config is from Ubuntu 16.04's kernel 4.4.0)
  • Docker 1.12.2
  • Storage driver is overlay2 (tested both XFS and ext4 as a backing filesystem)
$ docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 1.12.2
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.8.1-ldlk+
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.613 GiB
Name: tmp01
ID: JMBJ:3DS5:TXXR:H3NW:W2VB:36VO:SCFM:LXZM:L4YD:VB2R:BUSE:HCXZ
Docker Root Dir: /mnt
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

So the issue seems specific to ELRepo's patches/configs?

I'll try to set up a CentOS7 + ELRepo kernel env later..
(update: yes, I confirmed it is reproducible on CentOS7 + ELRepo kernel 4.8.1-1)

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Oct 16, 2016

Member

On CentOS7 + ELRepo kernel 4.8.1-1, I can reproduce the issue with xfs but not with ext4

Member

AkihiroSuda commented Oct 16, 2016

On CentOS7 + ELRepo kernel 4.8.1-1, I can reproduce the issue with xfs but not with ext4

@darrenscerri

This comment has been minimized.

Show comment
Hide comment
@darrenscerri

darrenscerri Oct 16, 2016

@thaJeztah I managed to replicate also using overlay using 3.18 kernel.

After further investigation, I noticed that my xfs partition had ftype set to 0. Red Hat states that xfs partitions need to be formatted with ftype set to 1 if they are to be used with overlay.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

I guess the issue can now be closed as it's not Docker related. Hopefully people who encounter this issue can stumble upon this issue for a solution.

darrenscerri commented Oct 16, 2016

@thaJeztah I managed to replicate also using overlay using 3.18 kernel.

After further investigation, I noticed that my xfs partition had ftype set to 0. Red Hat states that xfs partitions need to be formatted with ftype set to 1 if they are to be used with overlay.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

I guess the issue can now be closed as it's not Docker related. Hopefully people who encounter this issue can stumble upon this issue for a solution.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 16, 2016

Member

@darrenscerri oh, interesting; would this be something we should check for in our check-config script? https://github.com/docker/docker/blob/master/contrib/check-config.sh (Or from docker itself even when using the overlay driver) not sure if it's easy to add a check for this

Member

thaJeztah commented Oct 16, 2016

@darrenscerri oh, interesting; would this be something we should check for in our check-config script? https://github.com/docker/docker/blob/master/contrib/check-config.sh (Or from docker itself even when using the overlay driver) not sure if it's easy to add a check for this

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Oct 17, 2016

Member

rkt seems detecting this: https://github.com/coreos/rkt/pull/3105/files#diff-b5ccff3f3777d1b87c84277733a8c04fR393

I'll try to make a PR in similar way

Also, recent kernel shows a warning (not an error) in dmesg about this issue torvalds/linux@e7c0b59

Member

AkihiroSuda commented Oct 17, 2016

rkt seems detecting this: https://github.com/coreos/rkt/pull/3105/files#diff-b5ccff3f3777d1b87c84277733a8c04fR393

I'll try to make a PR in similar way

Also, recent kernel shows a warning (not an error) in dmesg about this issue torvalds/linux@e7c0b59

AkihiroSuda added a commit to AkihiroSuda/issues-docker that referenced this issue Oct 17, 2016

@vieux vieux closed this in #27433 Nov 9, 2016

AkihiroSuda added a commit to AkihiroSuda/issues-docker that referenced this issue Dec 5, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment