Can't upgrade/uninstall pip packages in Dockerfile

[skrech]

Hello,

Since recently our team is migrating to Concourse. I'm using oci-build-task to build my images and noticed that some of the builds are failing when trying to upgrade some of the pip packages in two different layers of a Dockerfile. Fortunately, it's easy to reproduce with dockerfile like this one:

FROM ubuntu:bionic

RUN apt-get update && apt-get install -y python3-pip python-tox

RUN pip3 install --upgrade pip

# RUN pip3 install --upgrade six
RUN pip3 uninstall -y six

WORKDIR /opt/testdir

CMD ["sh", "-exc" "'echo works'"]

Here "six" package is installed firstly by python-tox and then I need to upgrade it in subsequent step. However, intentionally I've commented the upgrade step and replaced it with uninstall -- this way we can clearly see where is the problem. With this setup the error message is:

#6 [3/5] RUN pip3 install --upgrade pip
#6 2.463 Collecting pip
#6 2.578   Downloading https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl (1.5MB)
#6 2.899 Installing collected packages: pip
#6 2.900   Found existing installation: pip 9.0.1
#6 2.903     Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
#6 4.102 Successfully installed pip-20.1.1
#6 DONE 4.4s

#7 [4/5] RUN pip3 uninstall -y six
#7 1.026 Found existing installation: six 1.11.0
#7 1.028 Uninstalling six-1.11.0:
#7 1.423 ERROR: Exception:
#7 1.423 Traceback (most recent call last):
#7 1.423   File "/usr/lib/python3.6/shutil.py", line 550, in move
#7 1.423     os.rename(src, real_dst)
#7 1.423 OSError: [Errno 18] Invalid cross-device link: '/usr/lib/python3/dist-packages/six-1.11.0.egg-info' -> '/usr/lib/python3/dist-packages/~ix-1.11.0.egg-info'
#7 1.423
#7 1.423 During handling of the above exception, another exception occurred:
#7 1.423
#7 1.423 Traceback (most recent call last):
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
#7 1.423     status = self.run(options, args)
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/uninstall.py", line 86, in run
#7 1.423     auto_confirm=options.yes, verbose=self.verbosity > 0,
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 676, in uninstall
#7 1.423     uninstalled_pathset.remove(auto_confirm, verbose)
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_uninstall.py", line 394, in remove
#7 1.423     moved.stash(path)
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_uninstall.py", line 283, in stash
#7 1.423     renames(path, new_path)
#7 1.423   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/misc.py", line 349, in renames
#7 1.423     shutil.move(old, new)
#7 1.423   File "/usr/lib/python3.6/shutil.py", line 562, in move
#7 1.423     rmtree(src)
#7 1.423   File "/usr/lib/python3.6/shutil.py", line 490, in rmtree
#7 1.423     onerror(os.rmdir, path, sys.exc_info())
#7 1.423   File "/usr/lib/python3.6/shutil.py", line 488, in rmtree
#7 1.423     os.rmdir(path)
#7 1.423 OSError: [Errno 22] Invalid argument: '/usr/lib/python3/dist-packages/six-1.11.0.egg-info'
#7 ERROR: executor failed running [/bin/sh -c pip3 uninstall -y six]: buildkit-runc did not terminate successfully
------
 > [4/5] RUN pip3 uninstall -y six:
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pip3 uninstall -y six]: buildkit-runc did not terminate successfully
FATA[0134] failed to build: build: exit status 1
FATA[0134] failed to run task: exit status 1
failed

I start the project on Concourse 5.8 with the following command:
fly -t rw execute -c test.yaml -i code="." --privileged

With the following test.yaml in the same dir as the Dockerfile:

---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: vito/oci-build-task

inputs:
  - name: code

outputs:
  - name: image

caches:
  - path: cache

params:
  CONTEXT: "./code"

run:
  path: build

---

The version installed from apt-get is pip-9.. which is working fine, however it's now deprecated. Our internal artifactory server insists on newer version of pip. However, with every version >9.. the previous error is displayed.

The interesting thin is that I tried on my Mac to build this image:
docker build -t test .
And with Buildkit as well:
DOCKER_BUILDKIT=1 docker build -t test .

Both of these commands succeed on the Mac.

Moreover, this Docker file is building fine on Concourse with DCinD (this task: https://github.com/meAmidos/dcind).

That being said, I suspect that the problem is in 'pip'! There is even a bug report opened on github for this last year:
pypa/pip#6943

However, the reason I'm writing the bug report here is because I don't really know what is going on under the hood either in oci-build-task or Concourse. Or maybe Buildkit is confusing something? Or maybe Buildkit in docker container is complicating stuff? The thing is that these docker image builds are not working only in this configuration: Concourse + oci-build-task. Apparently, python/pip is seeing the specific file located on different FS.

Regards,
Kristiyan

[vito]

Strange. I wonder if this issue is related?: docker/for-linux#480

What Linux kernel version are you running?

What does this output on the worker?:

sudo cat /sys/module/overlay/parameters/metacopy

If that prints Y, you could try echoing N to it and running again:

echo N | sudo tee /sys/module/overlay/parameters/metacopy

[skrech]

Hey @vito,

Linux version of what? I'm using your vito/oci-build-task container.
How can I connect to the worker? I don't own the Concourse installation.

The only thing that I can get is this output when building the Dockerfile:

#7 [4/7] RUN uname -a
#7 0.358 Linux buildkitsandbox 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
#7 DONE 0.8s

[vito]

Linux version of what? I'm using your vito/oci-build-task container.

Containers don't determine the Linux version - that's up to your worker machine's kernel which is running all the containers.

The only thing that I can get is this output when building the Dockerfile:

Thanks, that works! Looks like you're running 4.4, which is before metacopy was even introduced, so those commands won't help.

Unfortunately that was my only bright idea. This task is really just a wrapper around BuildKit, so I'm not really sure if there's anything I can do to fix this myself. I just pushed v0.4.0 which bumps BuildKit to v0.7.2, so I guess you could try that to see if anything changes.

[vito]

For what it's worth, that kernel version is over 4 years old. Would it be possible to upgrade? The latest LTS release is 5.4.

[skrech]

Hello,

I've tried with our infrastructure running Concourse 6.4 with workers featuring kernel 4.15 and there the problem seems to be non-existent. So, we can close this issue. Thanks for the help! :)

Regards,
Kristiyan

[vito]

@skrech Awesome, good to know - closing!