New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overlayfs whiteout files appearing within container #1097

Closed
johnl opened this Issue Jan 26, 2016 · 7 comments

Comments

Projects
None yet
5 participants
@johnl

johnl commented Jan 26, 2016

Running CoreOS stable (835.11.0), fresh install, ext4+overlayfs not btrfs.

installing a python package as part of an build builds results in "OSError: [Errno 2] No such file or directory" during the install process:

OSError: [Errno 2] No such file or directory: '/tmp/pip-IoDBTh-uninstall/usr/lib/python2.7/site-packages/pkg_resources.py'

that filename can be seen from the directory listing, but cannot be read or statted directly:

[root@d3b7c0d3b6ee /]# ls /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/ -lah
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/pkg_resources.py: No such file or directory
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/pkg_resources.pyc: No such file or directory
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/easy_install.pyc: No such file or directory
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/easy_install.pyo: No such file or directory
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/easy_install.py: No such file or directory
ls: cannot access /tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/pkg_resources.pyo: No such file or directory
total 16K
d-w-------. 2 root root 4.0K Jan 26 13:57 .
drwxr-xr-x. 3 root root 4.0K Jan 26 13:57 ..
c?????????? ? ?    ?       ?            ? easy_install.py
c?????????? ? ?    ?       ?            ? easy_install.pyc
c?????????? ? ?    ?       ?            ? easy_install.pyo
c?????????? ? ?    ?       ?            ? pkg_resources.py
c?????????? ? ?    ?       ?            ? pkg_resources.pyc
c?????????? ? ?    ?       ?            ? pkg_resources.pyo

Looking at that directory on the container layer directly on the host, the files exist and are overlayfs whiteout files:

srv-mqnaj overlay # ls d3b7c0d3b6ee2bcb243e3638e7ab4f699078bdf2c5de11b4eb4e3880e12a8ce2/upper/tmp/pip-QBYemC-uninstall/usr/lib/python2.7/site-packages/ -lah
total 40K
d-w------- 2 root root 4.0K Jan 26 13:57 .
drwxr-xr-x 3 root root 4.0K Jan 26 13:57 ..
c--------- 1 root root 0, 0 Jan 26 13:57 easy_install.py
c--------- 1 root root 0, 0 Jan 26 13:57 easy_install.pyc
c--------- 1 root root 0, 0 Jan 26 13:57 easy_install.pyo
c--------- 1 root root 0, 0 Jan 26 13:57 pkg_resources.py
c--------- 1 root root 0, 0 Jan 26 13:57 pkg_resources.pyc
c--------- 1 root root 0, 0 Jan 26 13:57 pkg_resources.pyo

I can reproduce this consistently by building this Dockerfile, and have reproduced it on two different hosts (both 835.11.0).

FROM centos:7
RUN yum update -y && yum -y install python-setuptools
RUN easy_install pip
RUN pip install --upgrade setuptools

And the same with an equivalent ubuntu version:

FROM ubuntu:trusty
RUN apt-get update && apt-get install -qy python-setuptools
RUN easy_install pip
RUN pip install --upgrade setuptools

I've not narrowed it down any further yet.

@NeilW

This comment has been minimized.

NeilW commented Jan 26, 2016

I've run the Dockerfile on the latest CoreOS Alpha stream (935.0.0) with the same result.

@johnl

This comment has been minimized.

johnl commented Jan 26, 2016

If I run the commands all in one shot, so that intermediary images are not created, the build succeeds without error:

FROM ubuntu:trusty
RUN apt-get update && apt-get install -qy python-setuptools && easy_install pip && pip install --upgrade setuptools

this works too, which creates one intermediary after apt-get update:

FROM ubuntu:trusty
RUN apt-get update
RUN apt-get install -qy python-setuptools && easy_install pip && pip install --upgrade setuptools

However, this does not work:

FROM ubuntu:trusty
RUN apt-get update && apt-get install -qy python-setuptools
RUN easy_install pip && pip install --upgrade setuptools

curious!

@johnl

This comment has been minimized.

johnl commented Jan 26, 2016

srv-9lo1a overlaytest # docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Fri Jan 22 06:07:01 UTC 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   cedd534-dirty
 Built:        Fri Jan 22 06:07:01 UTC 2016
 OS/Arch:      linux/amd64

srv-9lo1a overlaytest # docker info 
Containers: 9
Images: 52
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.2-coreos-r2
Operating System: CoreOS 835.11.0
CPUs: 2
Total Memory: 3.864 GiB
Name: srv-9lo1a.gb1.brightbox.com
ID: KA7V:4YG3:LRFN:D6EI:TI67:OBZG:KFXZ:FS7B:6V7L:KWXR:JWT4:B4YQ
@mjg59

This comment has been minimized.

mjg59 commented Jan 27, 2016

I'm seeing the same behaviour under Fedora 23, so this seems like an upstream overlayfs issue. We'll work with upstream to get this fixed as quickly as possible - I'm sorry for the inconvenience caused.

@mischief

This comment has been minimized.

mischief commented Mar 18, 2016

i'd be interested to know if this still fails in CoreOS 991.0.0, whose kernel has some overlay changes.

@johnl

This comment has been minimized.

johnl commented Mar 20, 2016

This was still broken in 983.0.0 but does indeed appear to be fixed in 991.0.0. I can no longer reproduce it with any of the examples above. Nice one, thanks.

@mischief

This comment has been minimized.

mischief commented Mar 20, 2016

fantastic :-) thanks for writing back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment