Docker build that fails with overlay storage backend only? #12327

lvangool · 2015-04-13T10:27:56Z

This Dockerfile doesn't seem to build with the overlay storage backend:

from ubuntu:trusty
RUN apt-get update && apt-get -y upgrade
RUN apt-get install -y build-essential g++ libblas-dev liblapack-dev gfortran python-dev python-pip
RUN pip install --upgrade numpy
RUN pip install --upgrade statsmodels
RUN pip install --upgrade flask

It ends up spitting out the below errors:

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1443, in install
    requirement.commit_uninstall()
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 610, in commit_uninstall
    self.uninstalled.commit()
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1860, in commit
    rmtree(self.save_dir)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 43, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/pip-dXgb99-uninstall/usr/lib/python2.7/dist-packages'

The same Dockerfile works with both AUFS and DeviceMapper storage backends (have not tried with BTRFS).

docker version:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

docker info:

Debug mode (server): false
Debug mode (client): true
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.18.0-031800-generic
Fds: 10
Goroutines: 15
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support

uname -a:

Linux my_server 3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Any suggestions or additional info I can provide?

The text was updated successfully, but these errors were encountered:

cpuguy83 · 2015-04-13T13:53:59Z

Can you try with either the master build or the latest 1.6 RC?
https://master.docker.com for master binary
#11635 (comment) for 1.6 RC

lvangool · 2015-04-13T17:07:40Z

Tested, exact same problem I'm afraid.

docker version:

Client version: 1.6.0-rc5
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): fc4825d
OS/Arch (client): linux/amd64
Server version: 1.6.0-rc5
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): fc4825d
OS/Arch (server): linux/amd64

LK4D4 · 2015-04-13T17:50:27Z

Yup, I can confirm in master.

vbatts · 2015-04-14T01:56:31Z

@LK4D4 what kernel version are you running and underlying filesystem (ext4?)?

lvangool · 2015-04-14T16:28:39Z

@vbatts yes, ext4 (kernel was 3.18 - tried with 3.19 too)

LK4D4 · 2015-04-14T16:29:45Z

@vbatts mine is 4.0 with ext4

rochacon · 2015-04-29T00:31:03Z

Not sure if this is related with the overlay driver. Installing pip with the get-pip.py script, which installs the latest version, works fine on overlay.

docker info

% docker info
Containers: 6
Images: 45
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 3.19.0
Operating System: CoreOS 633.1.0
CPUs: 1
Total Memory: 996.4 MiB

fail

FROM ubuntu:14.04
RUN apt-get update && apt-get install -yq python-pip && apt-get clean
RUN pip install -U pip

% docker build -t pip-on-docker-test .
Sending build context to Docker daemon  12.8 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> b7cf8f0d9e82
Step 1 : RUN apt-get update && apt-get install -yq python-pip && apt-get clean
 ---> Running in 296e47de116e
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg

...Ommited apt-get output...

Processing triggers for ca-certificates (20141019ubuntu0.14.04.1) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
 ---> d10860df49b9
Removing intermediate container 296e47de116e
Step 2 : RUN pip install -U pip
 ---> Running in 888281f1cea1
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-6.1.1-py2.py3-none-any.whl#md5=172eb5abab25a5e0f7a7b63c7a49378d
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1443, in install
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 610, in commit_uninstall
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1860, in commit
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 43, in rmtree
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib/python2.7/shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 254, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/pip-ihvxTk-uninstall/usr/bin'

Storing debug log for failure in /root/.pip/pip.log
INFO[0079] The command [/bin/sh -c pip install -U pip] returned a non-zero code: 2

success

FROM ubuntu:14.04
RUN apt-get update && apt-get install -yq curl python-dev && apt-get clean
RUN curl https://bootstrap.pypa.io/get-pip.py | python
RUN pip install -U pip

% docker build -t pip-on-docker-test .
Sending build context to Docker daemon 13.82 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
 ---> b7cf8f0d9e82
Step 1 : RUN apt-get update && apt-get install -yq curl python-dev && apt-get clean
 ---> Running in ef5a019b1a59
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg

...Ommited apt-get output...

Processing triggers for ca-certificates (20141019ubuntu0.14.04.1) ...
Updating certificates in /etc/ssl/certs... 173 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
 ---> b0da48adf546
Removing intermediate container ef5a019b1a59
Step 2 : RUN curl https://bootstrap.pypa.io/get-pip.py | python
 ---> Running in cce7df197a0b
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1387k  100 1387k    0     0  4294k      0 --:--:-- --:--:-- --:--:-- 4282k
/tmp/tmphICD6N/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
Collecting pip
/tmp/tmphICD6N/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  Downloading pip-6.1.1-py2.py3-none-any.whl (1.1MB)
Collecting setuptools
  Downloading setuptools-15.2-py2.py3-none-any.whl (501kB)
Installing collected packages: pip, setuptools
Successfully installed pip-6.1.1 setuptools-15.2
 ---> 7eb57b6961c5
Removing intermediate container cce7df197a0b
Step 3 : RUN pip install -U pip
 ---> Running in 5221ea38472c
Requirement already up-to-date: pip in /usr/local/lib/python2.7/dist-packages
 ---> f3670b07fff6
Removing intermediate container 5221ea38472c
Successfully built f3670b07fff6

vbatts · 2015-04-29T17:23:18Z

that would be nice to confirm. From my investigation, it seemed pretty murky in the failed example, that it made one of the egg info as a unix socket and failed to remove it from /tmp. Seemed like a bad file check somewhere, but I didn't feel like that was happening in the kernel...

LK4D4 · 2015-04-29T17:24:14Z

@vbatts unix-socket also mentioned in #12080

vbatts · 2015-04-29T17:28:54Z

correct. Overlayfs and unix sockets do not play nice together yet. But why
would a pip install make an egg info as a socket? those egg info are
usually only regular file or directory.

On Wed, Apr 29, 2015 at 1:24 PM, Alexander Morozov <notifications@github.com

wrote:

@vbatts https://github.com/vbatts unix-socket also mentioned in #12080
#12080

—
Reply to this email directly or view it on GitHub
#12327 (comment).

dustinlacewell · 2015-08-05T18:37:19Z

Suffering from this exactly as above with Ubuntu 14.04 with 3.19.0-25-generic and Docker 1.7.1

ghost · 2015-10-06T00:09:10Z

I have this too with Ubuntu 15.04 with 3.19.0-28-generic, Docker 1.8.2 configured with overlayfs:


Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3
 Built:        Sun Sep 13 08:20:28 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:21:21 UTC 2015
 OS/Arch:      linux/amd64

Containers: 4
Images: 185
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-28-generic
Operating System: Ubuntu 15.04
CPUs: 4
Total Memory: 11.72 GiB
Name: lux15
ID: GE62:5TVK:2MAB:KGHS:KMXH:RHAH:XSHQ:ZRTS:GIOG:O3TD:UGZ7:NG6Y
WARNING: No swap limit support

I have a second machine on which I configured the same OS (it's a VM - I just copied it) with Docker using aufs. The first one fails, the second works. The error line occurs during a pip install as above:

 OSError: [Errno 39] Directory not empty: '/tmp/pip-LpFx0F-uninstall/usr/local/lib/python2.7/dist-packages/nose/plugins'

zamf · 2015-10-07T16:07:12Z

Same problem here with 3.19.0-22-generic on Ubuntu 14.04 and


Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:19:00 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0a8c2e3
 Built:        Thu Sep 10 19:19:00 UTC 2015
 OS/Arch:      linux/amd64

Containers: 4
Images: 360
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-22-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 48
Total Memory: 503.9 GiB
Name: luke2
ID: HCWS:FZUK:Q23O:45EU:HVJW:ST6O:BMW7:HSCA:GAYK:UVPM:5BAN:LQO4
WARNING: No swap limit support

LK4D4 · 2015-10-15T21:26:02Z

@vbatts @rochacon @lvangool I can confirm that patch-series from https://lkml.org/lkml/2015/10/9/620 fixes both issues.

Overlay storage driver used by Deis has a bug and you cannot replace the debian installed six==1.8.0 with a newer version. This breaks `git push` to Deis to build the image with the builder. Make the requirements use six==1.8.0 as a workaround.

xdays · 2016-01-12T05:08:24Z

Hi @LK4D4 when will the patch being merged, I've updated my kernel to 4.4.0, and the problem still there.

[www:~]# docker info
Containers: 1
Images: 7
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.4.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 992.1 MiB
Name: www.xdays.me
ID: Q2NV:RAJE:LVY5:YZ4C:ZMUH:NJU2:6MZE:PFAT:XQRE:TAKN:2Y33:OSPI
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

grahamc · 2016-01-12T17:26:56Z

I have a similar issue, copied (and closed) from #19262:

The following Dockerfile fails to build on Overlay, but works on AUFS. I replicated this on two machines, docker-machine built and one running ubuntu trusty with linux kernel 4.2.

FROM python:3.4
RUN pip install flake8==2.5.1
RUN pip install flake8==2.5.0

Note it also fails with python:3.5, however changing it to the following works fine:

FROM python:3.4
RUN pip install flake8==2.5.1 && pip install flake8==2.5.0

Note: I know installing flake8 then installing a lesser version isn't great, but that doesn't change the fact that this is a bug. The problem is only exhibited on overlay, and not any other storage driver I've tried.

Build output on overlay:

➜  overlaybug  docker build --pull --no-cache  -t overlaybug .
Sending build context to Docker daemon 8.704 kB
Step 1 : FROM python:3.4
3.4: Pulling from library/python
523ef1d23f22: Pull complete
140f9bdfeb97: Pull complete
5c63804eac90: Pull complete
ce2b29af7753: Pull complete
5c2bdca41b86: Pull complete
b48f4076b06d: Pull complete
d6788afe8670: Pull complete
a81a1086b672: Pull complete
15b1f796b9f0: Pull complete
1efcb21adee7: Pull complete
d1dcef8ab76f: Pull complete
dd40266492e1: Pull complete
285252a442bf: Pull complete
Digest: sha256:19e0ba3b85f69233ab0d06954a58afa2672f659c6cb9bf86807d292c417a8dea
Status: Downloaded newer image for python:3.4
 ---> 285252a442bf
Step 2 : RUN pip install flake8==2.5.1
 ---> Running in 945cdf613f1c
Collecting flake8==2.5.1
  Downloading flake8-2.5.1-py2.py3-none-any.whl
Collecting pyflakes<1.1,>=0.8.1 (from flake8==2.5.1)
  Downloading pyflakes-1.0.0-py2.py3-none-any.whl (152kB)
Collecting pep8!=1.6.0,!=1.6.1,!=1.6.2,>=1.5.7 (from flake8==2.5.1)
  Downloading pep8-1.5.7-py2.py3-none-any.whl
Collecting mccabe<0.4,>=0.2.1 (from flake8==2.5.1)
  Downloading mccabe-0.3.1-py2.py3-none-any.whl
Installing collected packages: pyflakes, pep8, mccabe, flake8
Successfully installed flake8-2.5.1 mccabe-0.3.1 pep8-1.5.7 pyflakes-1.0.0
 ---> 25d407bf5a85
Removing intermediate container 945cdf613f1c
Step 3 : RUN pip install flake8==2.5.0
 ---> Running in 228cd950e051
Collecting flake8==2.5.0
  Downloading flake8-2.5.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): mccabe<0.4,>=0.2.1 in /usr/local/lib/python3.4/site-packages (from flake8==2.5.0)
Requirement already satisfied (use --upgrade to upgrade): pyflakes<1.1,>=0.8.1 in /usr/local/lib/python3.4/site-packages (from flake8==2.5.0)
Requirement already satisfied (use --upgrade to upgrade): pep8!=1.6.0,!=1.6.1,!=1.6.2,>=1.5.7 in /usr/local/lib/python3.4/site-packages (from flake8==2.5.0)
Installing collected packages: flake8
  Found existing installation: flake8 2.5.1
    Uninstalling flake8-2.5.1:
      Successfully uninstalled flake8-2.5.1
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/shutil.py", line 424, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'flake8'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/site-packages/pip/commands/install.py", line 311, in run
    root=options.root_path,
  File "/usr/local/lib/python3.4/site-packages/pip/req/req_set.py", line 657, in install
    requirement.commit_uninstall()
  File "/usr/local/lib/python3.4/site-packages/pip/req/req_install.py", line 729, in commit_uninstall
    self.uninstalled.commit()
  File "/usr/local/lib/python3.4/site-packages/pip/req/req_uninstall.py", line 152, in commit
    rmtree(self.save_dir)
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/retrying.py", line 212, in call
    raise attempt.get()
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/six.py", line 659, in reraise
    raise value
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/local/lib/python3.4/site-packages/pip/utils/__init__.py", line 90, in rmtree
    onerror=rmtree_errorhandler)
  File "/usr/local/lib/python3.4/shutil.py", line 468, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/lib/python3.4/shutil.py", line 406, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.4/shutil.py", line 406, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.4/shutil.py", line 406, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/local/lib/python3.4/shutil.py", line 426, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/local/lib/python3.4/site-packages/pip/utils/__init__.py", line 98, in rmtree_errorhandler
    if os.stat(path).st_mode & stat.S_IREAD:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-7imwu3vj-uninstall/usr/local/bin/flake8'
The command '/bin/sh -c pip install flake8==2.5.0' returned a non-zero code: 2

docker info:

➜  overlaybug  docker info
Containers: 16
Images: 37
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: overlaytest
ID: 4WVC:34GK:NLWL:3LFL:QIGK:J6LD:YZL5:DQ47:WL2J:XJRX:6MCZ:DQ6Z
Debug mode (server): true
 File Descriptors: 15
 Goroutines: 20
 System Time: 2016-01-12T16:47:52.96032768Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

docker version:

➜  overlaybug  docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5
 Built:        Sat Nov 21 00:49:19 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

created via:

docker-machine create --driver virtualbox --virtualbox-disk-size 5000 --engine-storage-driver overlay overlaytest

This also fails on another VM:

$ docker info
Containers: 7
Images: 154
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.0-23-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 16
Total Memory: 29.44 GiB
Name: ip-10-53-6-118
ID: 63LD:WH3H:LW4Y:S5FQ:UY5J:YMZE:U4WO:QPGE:A7L2:2JJW:CK2M:WYRF
Username: clarifyiodeploy
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

$ uname -a
Linux ip-10-53-6-118 4.2.0-23-generic #28~14.04.1-Ubuntu SMP Thu Dec 31 13:40:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

No interesting dmesg or docker.log output was found.

pkout · 2016-01-14T01:38:41Z

For those who don't know, until this gets merged, you can bypass this problem by running the pip upgrade command together with apt-get install to avoid file system layering. Something like this:

RUN apt-get install python-pip && pip install --upgrade pip

jgmize · 2016-02-23T07:29:56Z

I think this may be related to https://bugzilla.kernel.org/show_bug.cgi?id=109611.

@jgmize

Also fix the pip install with a workaround for moby/moby#12327. Thanks @jgmize!

Not sure whether this is related to moby/moby#19647..

steinnes · 2016-02-25T18:35:31Z

I had the same issue, my workaround was to simply install my own custom python packages in a virtualenvironment. That way pip never needed to attempt to uninstall any dist packages and the problem was worked around.

Here's my recipe for anyone interested and using ubuntu:14.04 as a base.

Before:

FROM ubuntu:14.04
MAINTAINER Takumi <steinn@takumi.com>
ENV DEBIAN_FRONTEND noninteractive
WORKDIR /app
RUN apt-get update && apt-get upgrade -y &&\
    apt-get install -y\
        curl wget htop make build-essential rsync cron anacron ntp python-dev python-pip git\
        python-software-properties software-properties-common libpq-dev libssl-dev libffi-dev\
        postgresql-client libzmq-dev libxml2-dev libxslt-dev libmagickwand-dev runit
RUN pip install requests virtualenv virtualenv-tools uwsgi==2.0.12
ADD requirements.txt /app/
ADD test-requirements.txt /app/
RUN pip install -r /app/test-requirements.txt
COPY . /app/
RUN pip install -r /app/test-requirements.txt
EXPOSE 5000
CMD uwsgi --yaml /app/uwsgi.yaml

After:

FROM ubuntu:14.04
MAINTAINER Takumi <steinn@takumi.com>
ENV DEBIAN_FRONTEND noninteractive
WORKDIR /app
RUN apt-get update && apt-get upgrade -y &&\
    apt-get install -y\
        curl wget htop make build-essential rsync cron anacron ntp python-dev python-pip git\
        python-software-properties software-properties-common libpq-dev libssl-dev libffi-dev\
        postgresql-client libzmq-dev libxml2-dev libxslt-dev libmagickwand-dev runit
RUN pip install requests virtualenv virtualenv-tools
RUN virtualenv /venv && /venv/bin/pip install --upgrade pip && /venv/bin/pip install uwsgi==2.0.12
ADD requirements.txt /app/
ADD test-requirements.txt /app/
RUN /venv/bin/pip install -r /app/test-requirements.txt
COPY . /app/
RUN /venv/bin/pip install -r /app/test-requirements.txt
EXPOSE 5000
CMD /venv/bin/uwsgi --yaml /app/uwsgi.yaml

As you can see the file is slightly messier, but the upside is that it works with my CI system now (codeship).

strohel · 2016-07-27T11:40:35Z

Hi all, I've recently upgraded Linux kernel to 4.6.4 (back from 4.1) and the problem seems to be gone! Do you have the same observation?

thaJeztah · 2016-07-27T11:50:03Z

According to https://bugzilla.kernel.org/show_bug.cgi?id=109611, this is fixed in kernel 4.5 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=45d11738969633ec07ca35d75d486bf2d8918df6

ping @LK4D4 can you verify if that's what solves this issue?

justincormack · 2016-09-19T22:20:13Z

Seems to be fixed in 4.4.21 too. (4.4 stable series has had a lot of the overlay fixes backported recently).

mkocikowski · 2016-10-03T17:01:14Z

This problem doesn't seem to be specific to python / pip. Running overlay fs on ext4, kernel 4.4 (info below) a simple COPY foo /foo will fail with "no such file or directory" error (error in the last line of the shell output below):

root@host:~# uname -a
Linux host 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@host:~# docker info
Containers: 25
 Running: 5
 Paused: 0
 Stopped: 20
Images: 28
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-38-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.795 GiB
Name: host
ID: SMFL:O7UP:7O4O:RVSD:QIQ2:FMX5:EVF5:UY74:336S:3QWI:DSEU:2HME
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
root@host:~# ls
Dockerfile  foo
root@host:~# cat Dockerfile
FROM alpine
COPY foo /foo
root@host:~# docker build .
Sending build context to Docker daemon  21.5 kB
Step 1 : FROM alpine
 ---> ee4603260daa
Step 2 : COPY foo /foo
stat /var/lib/docker/overlay/78c71862cccbd03b4a5be0f44e21753c7fee96ecc37f6e8048497d6f14141ee1/merged/foo: no such file or directory
root@host:~#

Strangely enough, restarting docker engine "solved" the problem.

thaJeztah · 2016-10-10T11:27:33Z

This issue is resolved in recent kernels

This issue is resolved in recent kernels; if possible, upgrade your kernel to a version that includes this fix, or contact your OS vendor to make sure they carry the fix that resolves this. For more information, read the comments above; #12327 (comment), and #12327 (comment).

Note that this is not a bug in Docker, so please don't comment that you're still having this on docker version x.y.z.

Addresses: Collecting pip Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB) Collecting wheel Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB) Installing collected packages: pip, wheel Found existing installation: pip 1.5.6 Uninstalling pip-1.5.6: Exception: Traceback (most recent call last): File "/tmp/tmpqu31Wp/pip.zip/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/tmp/tmpqu31Wp/pip.zip/pip/commands/install.py", line 342, in run prefix=options.prefix_path, File "/tmp/tmpqu31Wp/pip.zip/pip/req/req_set.py", line 778, in install requirement.uninstall(auto_confirm=True) File "/tmp/tmpqu31Wp/pip.zip/pip/req/req_install.py", line 754, in uninstall paths_to_remove.remove(auto_confirm) File "/tmp/tmpqu31Wp/pip.zip/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/tmp/tmpqu31Wp/pip.zip/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib/python2.7/shutil.py", line 300, in move rmtree(src) File "/usr/lib/python2.7/shutil.py", line 247, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib/python2.7/shutil.py", line 256, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib/python2.7/shutil.py", line 254, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/usr/lib/python2.7/dist-packages/pip/backwardcompat' when Docker is using the overlay fs backend. See: moby/moby#12327

See, moby/moby#12327 Travis error during selenium installation (only in the Python 3.6 test): https://travis-ci.org/webfp/tor-browser-selenium/jobs/310530961#L561

thaJeztah added area/storage/overlay kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. labels Apr 13, 2015

jessfraz added this to the 1.7.0 milestone Apr 13, 2015

LK4D4 mentioned this issue Apr 13, 2015

graphdriver: prefer prior driver state #11999

Merged

vbatts mentioned this issue Apr 14, 2015

graphdriver: promote overlay driver to first #12354

Closed

2 tasks

This was referenced May 25, 2015

"pip install" failed on overlay driver #13457

Closed

dind overlay backend: pip cleanup fails due to delete of non-empty directory #13411

Closed

tiborvass removed this from the 1.7.0 milestone May 28, 2015

thaJeztah mentioned this issue Dec 11, 2015

Build Error: Unable to build gem nokogiri from official ruby:onbuild image #18578

Closed

thaJeztah mentioned this issue Jan 10, 2016

overlay backend does not build the image #19215

Closed

bgruening mentioned this issue Jan 10, 2016

IE does not work bgruening/docker-galaxy-stable#127

Closed

grahamc mentioned this issue Jan 12, 2016

Dockerfile fails to build on overlay but works on aufs #19262

Closed

pmclanahan pushed a commit to pmclanahan/nucleus-sugar that referenced this issue Feb 23, 2016

Remove virtualenv

855188b

Also fix the pip install with a workaround for moby/moby#12327. Thanks @jgmize!

AkihiroSuda added a commit to AkihiroSuda/issues-docker that referenced this issue Feb 25, 2016

Add overlay issue(moby/moby#12327): pip fails

95af5f6

Not sure whether this is related to moby/moby#19647..

This was referenced Feb 26, 2016

Add Dockerfile for "git push deis" demos mozilla/bedrock#3906

Merged

Move from peep to pip8 mozilla/bedrock#3931

Merged

legal90 mentioned this issue Mar 31, 2016

Problem with files Parallels/docker-machine-parallels#30

Closed

icecrime added the version/1.5 label May 12, 2016

ssk2 added a commit to mesosphere-backup/dcos-jenkins-dind-agent that referenced this issue Jun 9, 2016

Upgrade Python tools in this step to mitigate: moby/moby#12327

56d307b

ssk2 mentioned this issue Jun 9, 2016

[DO NOT MERGE] Upgrade pip and update glibc package. mesosphere-backup/dcos-jenkins-dind-agent#4

Closed

AkihiroSuda added a commit to AkihiroSuda/issues-docker that referenced this issue Jul 28, 2016

Overlay pip issue moby/moby#12327 is mostly resolved

c9b8aa6

thaJeztah mentioned this issue Aug 4, 2016

os.rename throws exception when renaming directories on lower overlayfs layers #25409

Open

justincormack added the area/kernel label Sep 19, 2016

thaJeztah closed this as completed Oct 10, 2016

angeiv mentioned this issue Feb 14, 2017

Could not work by using image zabbix-web-*-mysql when using overlay storage driver zabbix/zabbix-docker#82

Closed

jmatsushita mentioned this issue Mar 31, 2017

edxapp prerequisite installation fails unexpectedly openedx-unsupported/devstack#72

Closed

thewtex mentioned this issue Jun 10, 2017

linux-*: Change common.docker to fix pip installation. dockcross/dockcross#151

Merged

TomasTomecek mentioned this issue Jun 14, 2017

openshift python module breaks build of conductor ansible/ansible-container#581

Closed

tglucas pushed a commit to tailucas/event-processor that referenced this issue May 8, 2020

Workaround for moby/moby#12327

64af2e0

tglucas pushed a commit to tailucas/remote-monitor that referenced this issue May 8, 2020

Workaround for moby/moby#12327

973a54f

wglambert mentioned this issue May 12, 2022

python:3.7-buster cannot upgrade pip docker-library/python#727

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker build that fails with overlay storage backend only? #12327

Docker build that fails with overlay storage backend only? #12327

lvangool commented Apr 13, 2015

cpuguy83 commented Apr 13, 2015

lvangool commented Apr 13, 2015

LK4D4 commented Apr 13, 2015

vbatts commented Apr 14, 2015

lvangool commented Apr 14, 2015

LK4D4 commented Apr 14, 2015

rochacon commented Apr 29, 2015

vbatts commented Apr 29, 2015

LK4D4 commented Apr 29, 2015

vbatts commented Apr 29, 2015

dustinlacewell commented Aug 5, 2015

ghost commented Oct 6, 2015

zamf commented Oct 7, 2015

LK4D4 commented Oct 15, 2015

xdays commented Jan 12, 2016

grahamc commented Jan 12, 2016

pkout commented Jan 14, 2016

jgmize commented Feb 23, 2016

steinnes commented Feb 25, 2016

strohel commented Jul 27, 2016

thaJeztah commented Jul 27, 2016

justincormack commented Sep 19, 2016 •

edited

mkocikowski commented Oct 3, 2016 •

edited

thaJeztah commented Oct 10, 2016

Docker build that fails with overlay storage backend only? #12327

Docker build that fails with overlay storage backend only? #12327

Comments

lvangool commented Apr 13, 2015

cpuguy83 commented Apr 13, 2015

lvangool commented Apr 13, 2015

LK4D4 commented Apr 13, 2015

vbatts commented Apr 14, 2015

lvangool commented Apr 14, 2015

LK4D4 commented Apr 14, 2015

rochacon commented Apr 29, 2015

docker info

fail

success

vbatts commented Apr 29, 2015

LK4D4 commented Apr 29, 2015

vbatts commented Apr 29, 2015

dustinlacewell commented Aug 5, 2015

ghost commented Oct 6, 2015

zamf commented Oct 7, 2015

LK4D4 commented Oct 15, 2015

xdays commented Jan 12, 2016

grahamc commented Jan 12, 2016

pkout commented Jan 14, 2016

jgmize commented Feb 23, 2016

steinnes commented Feb 25, 2016

strohel commented Jul 27, 2016

thaJeztah commented Jul 27, 2016

justincormack commented Sep 19, 2016 • edited

mkocikowski commented Oct 3, 2016 • edited

thaJeztah commented Oct 10, 2016

This issue is resolved in recent kernels

justincormack commented Sep 19, 2016 •

edited

mkocikowski commented Oct 3, 2016 •

edited