Unable to docker build on Linux kernel 4.19

[jetm]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Build without error

Actual behavior

When run docker build -t testing . this error appears on Linux kernel 4.19:

dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command '/bin/sh -c apt-get install -y locales &&     sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen &&     locale-gen' returned a non-zero code: 100

On Linux kernel v4.18 it works. I have tested docker v18.06 and v18.09.

Steps to reproduce the behavior

On a Linux kernel 4.19 use this Dockerfile:

FROM ubuntu:16.04

# Let the container know that there is no tty
ENV DEBIAN_FRONTEND noninteractive

RUN echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse" \
        > /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    apt-get update

# Ensure UTF-8 lang and locale
RUN apt-get install -y locales && \
    sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

I run docker build -t testing .

Output of docker version:

Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 32
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.19.1-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.41GiB
Name: javier-homepc
ID: 6ULE:QTFG:T6YJ:QIRD:URUL:GEZR:ZWUK:R2J2:NAWY:ALEY:PSSJ:EW4P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Further information

It's a physical machine with Manjaro Linux based on Arch Linux OS (docker is from Arch Linux and Linux kernel from Manjaro)

[thaJeztah]

Sounds like either an issue with overlayfs or dpkg not playing well with overlayfs.

A quick search brought me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211;msg=17

[jetm]

@thaJeztah: Thanks. It seems the bug it's still open: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211

[Omrigan]

There is an even simpler example:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg

Any fixes apart from downgrading kernel?

[ffernand]

I posted a comment here about what was discovered and a subsequent comment shows that the default for the overlay config metacopy was switched from N to Y in kernel 4.19...

The following should do the trick to get you going ...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

... remember that you'll likely have to restart your containers. The above isn't permanent, a reboot will reset the value back to Y, so add this in a script somewhere that gets called on boot.

You can also try /sbin/modprobe overlay metacopy=N though it's up to you when this is called (presumably before the module is loaded?)

You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works.

[jetm]

There is a fix for the kernel: https://www.spinics.net/lists/linux-unionfs/msg06316.html. Hope it's merged in 4.19.3.

[znmeb]

I'm seeing the same error using buildah bud to do the build. I'm on Antergos (Arch-based) so I can just drop back to the LTS kernel.

[znmeb]

@ffernand: "You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works."

I just tried it, and it does appear to be working. Antergos (Arch) Linux, in docker-compose build of two images based on Debian stretch.

[grski]

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

[dimitar-petrov]

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

Thanks!

[jbaumcloud]

I am still getting the same error:

...
Get:186 http://archive.ubuntu.com/ubuntu bionic/main amd64 vdpau-driver-all amd64 1.1.1-3ubuntu1 [4674 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 75.8 MB in 5s (15.1 MB/s)
Selecting previously unselected package multiarch-support.
(Reading database ... 4038 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.27-3ubuntu1_amd64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1) ...
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

on Kernel Version: 4.19.4-1-MANJARO when using overlay2 Storage Driver. When using aufs everything works fine for me.

[arthfl]

I am too still getting the error with 4.19.4-arch1-1-ARCH

[tachigami]

It`s solved after update kernel to 4.19.4

Isn't for me 4.19.4-1-MANJARO

[vcraescu]

Linux 4.19.4-1-MANJARO

still getting the error!

[rohitpaulk]

I initially found this on 4.19.1, upgrading to 4.19.4 didn't help. (4.19.4-arch1-1-ARCH)

[rohitpaulk]

(Running the command mentioned above did fix it)

echo N | sudo tee /sys/module/overlay/parameters/metacopy

[ffernand]

@GH0st3rs I also upgraded my kernel to 4.19.4-arch1-1-ARCH (archlinux) but the issue was not resolved for me when metacopy was turned on...

sudo cat /sys/module/overlay/parameters/metacopy
Y

Are you sure you don't have the option overlay.metacopy=N in your command line for the kernel (via grub) or maybe you have echo N | sudo tee /sys/module/overlay/parameters/metacopy in your bash/shell init script?

[ejprice]

The problem exists for me on 4.19.4-1-MANJARO with metacopy on.

[SalahAdDin]

It also exists in x86_64 Linux 4.19.4-arch1-1-ARCH.

[SalahAdDin]

@GH0st3rs It is a partial solution, it works for me right now, but of course, if i reboot the computer i will lose this fix. Waiting for a total solution.

[ejprice]

@SalahAdDin you can make an entry into the kernel command line to make this fix permanent.

edit /etc/defaults/grub and find the line that says:
GRUB_CMDLINE_LINUX=""

change this to:
GRUB_CMDLINE_LINUX="overlay.metacopy=N"

Then, rebuild your Grub config:
update-grub

You'll need root permissions to edit/run the commands so either preface the command with sudo or switch user to the root account. e.g.

sudo vi /etc/default/grub
sudo update-grub

And, of course, the tricky part is remembering you did this so you can remove it once this issue is resolved 😜 That will be the part I screw up.

[teohhanhui]

You should not change GRUB_CMDLINE_LINUX (it's used as a fallback), but add it to the end of GRUB_CMDLINE_LINUX_DEFAULT, e.g.

GRUB_CMDLINE_LINUX_DEFAULT="quiet overlay.metacopy=N"

[ejprice]

@teohhanhui I'd say that's just a matter of preference and not an absolute. The user is better off determining which is better given their application. That is why both options exist.

[teohhanhui]

From the manual:

‘GRUB_CMDLINE_LINUX’

Command-line arguments to add to menu entries for the Linux kernel.

‘GRUB_CMDLINE_LINUX_DEFAULT’

Unless ‘GRUB_DISABLE_RECOVERY’ is set to ‘true’, two menu entries will be generated for each Linux kernel: one default entry and one entry for recovery mode. This option lists command-line arguments to add only to the default menu entry, after those listed in ‘GRUB_CMDLINE_LINUX’.

https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html

[HubbeKing]

Issue does appear to be solved in 4.19.11-arch1-1-ARCH
Built this example without issue

There is an even simpler example:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg

Any fixes apart from downgrading kernel?

[loshz]

I can also confirm this is now working for me on 4.19.11-arch1-1-ARCH

[ax-slad]

Running into same issue, but proposed solution doesn't work for me as there is no such file as metacopy. Any suggestions on how to proceed?

root@12d2235747e7:/# ls -rtl /sys/module/overlay/parameters/
total 0
-rw-r--r-- 1 root root 4096 Jan 14 23:58 check_copy_up

Also, if I understand correctly, kernel version seems to be 4.9?

root@12d2235747e7:/# uname -r
4.9.125-linuxkit

[kolyshkin]

@ax-slad there is no such feature as metacopy in kernel 4.9, thus the issue you are having is something entirely different. Feel free to file a separate issue.

[rulatir]

Is the overlay.metacopy=N hack still needed as of the newest kernels? I am trying to clean up my kernel command line because it has accreted lots of hacks/fixes that may no longer be necessary...

[WhyNotHugo]

@rulatir I've recently removed it and everything's fine on my end.

[freeone3000]

I'm getting this issue on kernel version 5.4.0. systool -vm overlay shows that metacopy=N, but I'm still getting:

dpkg: error processing archive /tmp/apt-dpkg-install-yt98Qc/225-nvidia-utils-450_450.66-0ubuntu0.18.04.1_amd64.deb (--unpack):
unable to make backup link of './usr/bin/nvidia-debugdump' before installing new version: Invalid cross-device link

[Tony763]

With kernel: 5.8.0-29-generic

dpkg: error processing archive /var/cache/apt/archives/tzdata_2019a-0ubuntu0.14.04_all.deb (--unpack):
 unable to make backup link of './usr/share/zoneinfo/UCT' before installing new version: Invalid cross-device link

[foresto]

I'm experiencing this on debian kernel 5.9.0-4-arm64.

docker.io package: 19.03.13+dfsg3-1 arm64

rootless mode

dockerd flags: --storage-driver=overlay --experimental

overlay module parameters:
metacopy=N
redirect_always_follow=Y
redirect_dir=Y or N

Dockerfile to reproduce:

FROM node:10-slim
RUN apt-get update; apt-get install -y git

[SalahAdDin]

@foresto Did you reboot your computer after updating the kernel?

[foresto]

@SalahAdDin Yes, I know that kernel updates require a reboot to take effect. That is not relevant to this bug in my experience. The bug occurs even after I apply the overlay module parameters I listed above, regardless of whether I apply them manually through sysfs or with a modprobe.d conf file (plus reboot).

[SalahAdDin]

@foresto Every time i get this bug it is because i must reboot the system before to launch Docker after updating the kernel.

[Patrick-Remy]

UPDATE: Also when using fuse-overlayfs, same error...

Issue still exists for me on Linux w1 5.10.0-0.bpo.7-amd64 #1 SMP Debian 5.10.40-1~bpo10+1 (2021-06-04) x86_64 using buildkit (0.7.2 and 0.8.3 affected)

 docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:47 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 119
  Running: 52
  Paused: 0
  Stopped: 67
 Images: 38
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.10.0-0.bpo.7-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.27GiB
 Name: w1
 ID: MEAN:IWDJ:LON2:C3PY:QUYF:5GGG:HBRF:Z4GM:WFCP:NL5U:7C2F:UCAZ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[whatisor]

I posted a comment here about what was discovered and a subsequent comment shows that the default for the overlay config metacopy was switched from N to Y in kernel 4.19...

The following should do the trick to get you going ...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

... remember that you'll likely have to restart your containers. The above isn't permanent, a reboot will reset the value back to Y, so add this in a script somewhere that gets called on boot.

You can also try /sbin/modprobe overlay metacopy=N though it's up to you when this is called (presumably before the module is loaded?)

You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works.

it does not work for me in ubuntu 20.04 of gcp instance

[whatisor]

Solution for someone who may need:

• Combine commands into single RUN