Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to docker build on Linux kernel 4.19 #480

Open
jetm opened this issue Nov 10, 2018 · 45 comments
Open

Unable to docker build on Linux kernel 4.19 #480

jetm opened this issue Nov 10, 2018 · 45 comments

Comments

@jetm
Copy link

@jetm jetm commented Nov 10, 2018

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Build without error

Actual behavior

When run docker build -t testing . this error appears on Linux kernel 4.19:

dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command '/bin/sh -c apt-get install -y locales &&     sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen &&     locale-gen' returned a non-zero code: 100

On Linux kernel v4.18 it works. I have tested docker v18.06 and v18.09.

Steps to reproduce the behavior

On a Linux kernel 4.19 use this Dockerfile:

FROM ubuntu:16.04

# Let the container know that there is no tty
ENV DEBIAN_FRONTEND noninteractive

RUN echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse" \
        > /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    apt-get update

# Ensure UTF-8 lang and locale
RUN apt-get install -y locales && \
    sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

I run docker build -t testing .

Output of docker version:

Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 32
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.19.1-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.41GiB
Name: javier-homepc
ID: 6ULE:QTFG:T6YJ:QIRD:URUL:GEZR:ZWUK:R2J2:NAWY:ALEY:PSSJ:EW4P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Further information

It's a physical machine with Manjaro Linux based on Arch Linux OS (docker is from Arch Linux and Linux kernel from Manjaro)

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 11, 2018

Sounds like either an issue with overlayfs or dpkg not playing well with overlayfs.

A quick search brought me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211;msg=17

@jetm
Copy link
Author

@jetm jetm commented Nov 11, 2018

@thaJeztah: Thanks. It seems the bug it's still open: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211

@Omrigan
Copy link

@Omrigan Omrigan commented Nov 14, 2018

There is an even simpler example:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg

Any fixes apart from downgrading kernel?

@ffernand
Copy link

@ffernand ffernand commented Nov 14, 2018

I posted a comment here about what was discovered and a subsequent comment shows that the default for the overlay config metacopy was switched from N to Y in kernel 4.19...

The following should do the trick to get you going ...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

... remember that you'll likely have to restart your containers. The above isn't permanent, a reboot will reset the value back to Y, so add this in a script somewhere that gets called on boot.

You can also try /sbin/modprobe overlay metacopy=N though it's up to you when this is called (presumably before the module is loaded?)

You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works.

@jetm
Copy link
Author

@jetm jetm commented Nov 15, 2018

There is a fix for the kernel: https://www.spinics.net/lists/linux-unionfs/msg06316.html. Hope it's merged in 4.19.3.

@znmeb
Copy link

@znmeb znmeb commented Nov 18, 2018

I'm seeing the same error using buildah bud to do the build. I'm on Antergos (Arch-based) so I can just drop back to the LTS kernel.

@znmeb
Copy link

@znmeb znmeb commented Nov 19, 2018

@ffernand: "You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works."

I just tried it, and it does appear to be working. Antergos (Arch) Linux, in docker-compose build of two images based on Debian stretch.

@grski
Copy link

@grski grski commented Nov 21, 2018

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

@dimitar-petrov
Copy link

@dimitar-petrov dimitar-petrov commented Nov 21, 2018

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

Thanks!

zenhack added a commit to zenhack/haskell-capnp that referenced this issue Nov 26, 2018
I'm trying to make some changes and running afoul of:

    docker/for-linux#480

GHC is easy enough to install on alpine, so rather than fight with the
Debian based Haskell images, I'll just switch to something that doesn't
have this problem.
@jbaumcloud
Copy link

@jbaumcloud jbaumcloud commented Nov 26, 2018

I am still getting the same error:

...
Get:186 http://archive.ubuntu.com/ubuntu bionic/main amd64 vdpau-driver-all amd64 1.1.1-3ubuntu1 [4674 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 75.8 MB in 5s (15.1 MB/s)
Selecting previously unselected package multiarch-support.
(Reading database ... 4038 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.27-3ubuntu1_amd64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1) ...
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

on Kernel Version: 4.19.4-1-MANJARO when using overlay2 Storage Driver. When using aufs everything works fine for me.

@arthfl
Copy link

@arthfl arthfl commented Nov 26, 2018

I am too still getting the error with 4.19.4-arch1-1-ARCH

@TeroBlaZe
Copy link

@TeroBlaZe TeroBlaZe commented Nov 26, 2018

It`s solved after update kernel to 4.19.4

Isn't for me 4.19.4-1-MANJARO

@vcraescu
Copy link

@vcraescu vcraescu commented Nov 26, 2018

Linux 4.19.4-1-MANJARO

still getting the error!

@rohitpaulk
Copy link

@rohitpaulk rohitpaulk commented Nov 26, 2018

I initially found this on 4.19.1, upgrading to 4.19.4 didn't help. (4.19.4-arch1-1-ARCH)

@rohitpaulk
Copy link

@rohitpaulk rohitpaulk commented Nov 26, 2018

(Running the command mentioned above did fix it)

echo N | sudo tee /sys/module/overlay/parameters/metacopy
@ffernand
Copy link

@ffernand ffernand commented Nov 26, 2018

@GH0st3rs I also upgraded my kernel to 4.19.4-arch1-1-ARCH (archlinux) but the issue was not resolved for me when metacopy was turned on...

sudo cat /sys/module/overlay/parameters/metacopy
Y

Are you sure you don't have the option overlay.metacopy=N in your command line for the kernel (via grub) or maybe you have echo N | sudo tee /sys/module/overlay/parameters/metacopy in your bash/shell init script?

@ejprice
Copy link

@ejprice ejprice commented Nov 27, 2018

The problem exists for me on 4.19.4-1-MANJARO with metacopy on.

@SalahAdDin
Copy link

@SalahAdDin SalahAdDin commented Nov 27, 2018

It also exists in x86_64 Linux 4.19.4-arch1-1-ARCH.

@SalahAdDin
Copy link

@SalahAdDin SalahAdDin commented Nov 27, 2018

@GH0st3rs It is a partial solution, it works for me right now, but of course, if i reboot the computer i will lose this fix. Waiting for a total solution.

@ejprice
Copy link

@ejprice ejprice commented Nov 27, 2018

@SalahAdDin you can make an entry into the kernel command line to make this fix permanent.

edit /etc/defaults/grub and find the line that says:
GRUB_CMDLINE_LINUX=""

change this to:
GRUB_CMDLINE_LINUX="overlay.metacopy=N"

Then, rebuild your Grub config:
update-grub

You'll need root permissions to edit/run the commands so either preface the command with sudo or switch user to the root account. e.g.

sudo vi /etc/default/grub
sudo update-grub

And, of course, the tricky part is remembering you did this so you can remove it once this issue is resolved 😜 That will be the part I screw up.

@teohhanhui
Copy link

@teohhanhui teohhanhui commented Nov 27, 2018

You should not change GRUB_CMDLINE_LINUX (it's used as a fallback), but add it to the end of GRUB_CMDLINE_LINUX_DEFAULT, e.g.

GRUB_CMDLINE_LINUX_DEFAULT="quiet overlay.metacopy=N"
@ejprice
Copy link

@ejprice ejprice commented Nov 27, 2018

@teohhanhui I'd say that's just a matter of preference and not an absolute. The user is better off determining which is better given their application. That is why both options exist.

@teohhanhui
Copy link

@teohhanhui teohhanhui commented Nov 27, 2018

From the manual:

‘GRUB_CMDLINE_LINUX’

Command-line arguments to add to menu entries for the Linux kernel.

‘GRUB_CMDLINE_LINUX_DEFAULT’

Unless ‘GRUB_DISABLE_RECOVERY’ is set to ‘true’, two menu entries will be generated for each Linux kernel: one default entry and one entry for recovery mode. This option lists command-line arguments to add only to the default menu entry, after those listed in ‘GRUB_CMDLINE_LINUX’.

https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html

@NamPNQ
Copy link

@NamPNQ NamPNQ commented Nov 30, 2018

Workaround for waiting next docker release
$ echo N | sudo tee /sys/module/overlay/parameters/metacopy
stop running container -> start container (restart does not work correctly)
ensure docker mount overlayfs not use metacopy=on
$ mount | grep "docker"

@rtizzy
Copy link

@rtizzy rtizzy commented Dec 12, 2018

As other mentioned, this occurs on later versions of 4.19

Linux faithful 4.19.6-1-MANJARO

The mentioned echo N | sudo tee /sys/module/overlay/parameters/metacopy does resolve the issue.

@saro
Copy link

@saro saro commented Dec 14, 2018

Issue is still present in Archlinux kernel 4.19.8-arch1-1-ARCH

@HubbeKing
Copy link

@HubbeKing HubbeKing commented Dec 20, 2018

Looks like the kernel fix is merged in starting from version 4.19.11
https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.11

@philidor-green
Copy link

@philidor-green philidor-green commented Dec 20, 2018

Still exist 4.19.9-arch1-1-ARCH

@kolyshkin
Copy link

@kolyshkin kolyshkin commented Dec 20, 2018

Still exist 4.19.9-arch1-1-ARCH

@fishbullet as pointed out above, the patch appeared in 4.19.11 (released a day ago) so you need to wait for it to appear in Arch

@HubbeKing
Copy link

@HubbeKing HubbeKing commented Dec 21, 2018

Issue does appear to be solved in 4.19.11-arch1-1-ARCH
Built this example without issue

There is an even simpler example:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg

Any fixes apart from downgrading kernel?

@syscll
Copy link

@syscll syscll commented Dec 21, 2018

I can also confirm this is now working for me on 4.19.11-arch1-1-ARCH

@ax-slad
Copy link

@ax-slad ax-slad commented Jan 15, 2019

Running into same issue, but proposed solution doesn't work for me as there is no such file as metacopy. Any suggestions on how to proceed?

root@12d2235747e7:/# ls -rtl /sys/module/overlay/parameters/
total 0
-rw-r--r-- 1 root root 4096 Jan 14 23:58 check_copy_up

Also, if I understand correctly, kernel version seems to be 4.9?

root@12d2235747e7:/# uname -r
4.9.125-linuxkit

@kolyshkin
Copy link

@kolyshkin kolyshkin commented Jan 16, 2019

@ax-slad there is no such feature as metacopy in kernel 4.9, thus the issue you are having is something entirely different. Feel free to file a separate issue.

@rulatir
Copy link

@rulatir rulatir commented Mar 24, 2020

Is the overlay.metacopy=N hack still needed as of the newest kernels? I am trying to clean up my kernel command line because it has accreted lots of hacks/fixes that may no longer be necessary...

@WhyNotHugo
Copy link

@WhyNotHugo WhyNotHugo commented Mar 24, 2020

@rulatir I've recently removed it and everything's fine on my end.

@freeone3000
Copy link

@freeone3000 freeone3000 commented Sep 2, 2020

I'm getting this issue on kernel version 5.4.0. systool -vm overlay shows that metacopy=N, but I'm still getting:

dpkg: error processing archive /tmp/apt-dpkg-install-yt98Qc/225-nvidia-utils-450_450.66-0ubuntu0.18.04.1_amd64.deb (--unpack):
unable to make backup link of './usr/bin/nvidia-debugdump' before installing new version: Invalid cross-device link

@Tony763
Copy link

@Tony763 Tony763 commented Nov 22, 2020

With kernel: 5.8.0-29-generic

dpkg: error processing archive /var/cache/apt/archives/tzdata_2019a-0ubuntu0.14.04_all.deb (--unpack):
 unable to make backup link of './usr/share/zoneinfo/UCT' before installing new version: Invalid cross-device link

@foresto
Copy link

@foresto foresto commented Dec 1, 2020

I'm experiencing this on debian kernel 5.9.0-4-arm64.

docker.io package: 19.03.13+dfsg3-1 arm64

rootless mode

dockerd flags: --storage-driver=overlay --experimental

overlay module parameters:
metacopy=N
redirect_always_follow=Y
redirect_dir=Y or N

Dockerfile to reproduce:

FROM node:10-slim
RUN apt-get update; apt-get install -y git
@SalahAdDin
Copy link

@SalahAdDin SalahAdDin commented Dec 1, 2020

@foresto Did you reboot your computer after updating the kernel?

@foresto
Copy link

@foresto foresto commented Dec 1, 2020

@SalahAdDin Yes, I know that kernel updates require a reboot to take effect. That is not relevant to this bug in my experience. The bug occurs even after I apply the overlay module parameters I listed above, regardless of whether I apply them manually through sysfs or with a modprobe.d conf file (plus reboot).

@SalahAdDin
Copy link

@SalahAdDin SalahAdDin commented Dec 1, 2020

@foresto Every time i get this bug it is because i must reboot the system before to launch Docker after updating the kernel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.