New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to docker build on Linux kernel 4.19 #480

Open
jetm opened this Issue Nov 10, 2018 · 30 comments

Comments

Projects
None yet
@jetm

jetm commented Nov 10, 2018

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Build without error

Actual behavior

When run docker build -t testing . this error appears on Linux kernel 4.19:

dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command '/bin/sh -c apt-get install -y locales &&     sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen &&     locale-gen' returned a non-zero code: 100

On Linux kernel v4.18 it works. I have tested docker v18.06 and v18.09.

Steps to reproduce the behavior

On a Linux kernel 4.19 use this Dockerfile:

FROM ubuntu:16.04

# Let the container know that there is no tty
ENV DEBIAN_FRONTEND noninteractive

RUN echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse" \
        > /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse" \
        >> /etc/apt/sources.list && \
    apt-get update

# Ensure UTF-8 lang and locale
RUN apt-get install -y locales && \
    sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'

I run docker build -t testing .

Output of docker version:

Client:
 Version:           18.09.0-ce
 API version:       1.39
 Go version:        go1.11.2
 Git commit:        4d60db472b
 Built:             Fri Nov  9 00:05:34 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.2
  Git commit:       4d60db472b
  Built:            Fri Nov  9 00:05:11 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 32
Server Version: 18.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.19.1-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.41GiB
Name: javier-homepc
ID: 6ULE:QTFG:T6YJ:QIRD:URUL:GEZR:ZWUK:R2J2:NAWY:ALEY:PSSJ:EW4P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Further information

It's a physical machine with Manjaro Linux based on Arch Linux OS (docker is from Arch Linux and Linux kernel from Manjaro)

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Nov 11, 2018

Sounds like either an issue with overlayfs or dpkg not playing well with overlayfs.

A quick search brought me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211;msg=17

@jetm

This comment has been minimized.

jetm commented Nov 11, 2018

@thaJeztah: Thanks. It seems the bug it's still open: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211

@Omrigan

This comment has been minimized.

Omrigan commented Nov 14, 2018

There is an even simpler example:

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg

Any fixes apart from downgrading kernel?

@ffernand

This comment has been minimized.

ffernand commented Nov 14, 2018

I posted a comment here about what was discovered and a subsequent comment shows that the default for the overlay config metacopy was switched from N to Y in kernel 4.19...

The following should do the trick to get you going ...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

... remember that you'll likely have to restart your containers. The above isn't permanent, a reboot will reset the value back to Y, so add this in a script somewhere that gets called on boot.

You can also try /sbin/modprobe overlay metacopy=N though it's up to you when this is called (presumably before the module is loaded?)

You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works.

@jetm

This comment has been minimized.

jetm commented Nov 15, 2018

There is a fix for the kernel: https://www.spinics.net/lists/linux-unionfs/msg06316.html. Hope it's merged in 4.19.3.

@znmeb

This comment has been minimized.

znmeb commented Nov 18, 2018

I'm seeing the same error using buildah bud to do the build. I'm on Antergos (Arch-based) so I can just drop back to the LTS kernel.

@znmeb

This comment has been minimized.

znmeb commented Nov 19, 2018

@ffernand: "You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works."

I just tried it, and it does appear to be working. Antergos (Arch) Linux, in docker-compose build of two images based on Debian stretch.

@grski

This comment has been minimized.

grski commented Nov 21, 2018

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

@dimitar-petrov

This comment has been minimized.

dimitar-petrov commented Nov 21, 2018

what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro

Thanks!

zenhack added a commit to zenhack/haskell-capnp that referenced this issue Nov 26, 2018

Switch over to an alpine based CI image.
I'm trying to make some changes and running afoul of:

    docker/for-linux#480

GHC is easy enough to install on alpine, so rather than fight with the
Debian based Haskell images, I'll just switch to something that doesn't
have this problem.
@jbaumcloud

This comment has been minimized.

jbaumcloud commented Nov 26, 2018

I am still getting the same error:

...
Get:186 http://archive.ubuntu.com/ubuntu bionic/main amd64 vdpau-driver-all amd64 1.1.1-3ubuntu1 [4674 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 75.8 MB in 5s (15.1 MB/s)
Selecting previously unselected package multiarch-support.
(Reading database ... 4038 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.27-3ubuntu1_amd64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1) ...
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

on Kernel Version: 4.19.4-1-MANJARO when using overlay2 Storage Driver. When using aufs everything works fine for me.

@arthfl

This comment has been minimized.

arthfl commented Nov 26, 2018

I am too still getting the error with 4.19.4-arch1-1-ARCH

@TeroBlaZe

This comment has been minimized.

TeroBlaZe commented Nov 26, 2018

It`s solved after update kernel to 4.19.4

Isn't for me 4.19.4-1-MANJARO

@vcraescu

This comment has been minimized.

vcraescu commented Nov 26, 2018

Linux 4.19.4-1-MANJARO

still getting the error!

@rohitpaulk

This comment has been minimized.

rohitpaulk commented Nov 26, 2018

I initially found this on 4.19.1, upgrading to 4.19.4 didn't help. (4.19.4-arch1-1-ARCH)

@rohitpaulk

This comment has been minimized.

rohitpaulk commented Nov 26, 2018

(Running the command mentioned above did fix it)

echo N | sudo tee /sys/module/overlay/parameters/metacopy
@ffernand

This comment has been minimized.

ffernand commented Nov 26, 2018

@GH0st3rs I also upgraded my kernel to 4.19.4-arch1-1-ARCH (archlinux) but the issue was not resolved for me when metacopy was turned on...

sudo cat /sys/module/overlay/parameters/metacopy
Y

Are you sure you don't have the option overlay.metacopy=N in your command line for the kernel (via grub) or maybe you have echo N | sudo tee /sys/module/overlay/parameters/metacopy in your bash/shell init script?

@ejprice

This comment has been minimized.

ejprice commented Nov 27, 2018

The problem exists for me on 4.19.4-1-MANJARO with metacopy on.

@SalahAdDin

This comment has been minimized.

SalahAdDin commented Nov 27, 2018

It also exists in x86_64 Linux 4.19.4-arch1-1-ARCH.

@SalahAdDin

This comment has been minimized.

SalahAdDin commented Nov 27, 2018

@GH0st3rs It is a partial solution, it works for me right now, but of course, if i reboot the computer i will lose this fix. Waiting for a total solution.

@ejprice

This comment has been minimized.

ejprice commented Nov 27, 2018

@SalahAdDin you can make an entry into the kernel command line to make this fix permanent.

edit /etc/defaults/grub and find the line that says:
GRUB_CMDLINE_LINUX=""

change this to:
GRUB_CMDLINE_LINUX="overlay.metacopy=N"

Then, rebuild your Grub config:
update-grub

You'll need root permissions to edit/run the commands so either preface the command with sudo or switch user to the root account. e.g.

sudo vi /etc/default/grub
sudo update-grub

And, of course, the tricky part is remembering you did this so you can remove it once this issue is resolved 😜 That will be the part I screw up.

@teohhanhui

This comment has been minimized.

teohhanhui commented Nov 27, 2018

You should not change GRUB_CMDLINE_LINUX (it's used as a fallback), but add it to the end of GRUB_CMDLINE_LINUX_DEFAULT, e.g.

GRUB_CMDLINE_LINUX_DEFAULT="quiet overlay.metacopy=N"
@ejprice

This comment has been minimized.

ejprice commented Nov 27, 2018

@teohhanhui I'd say that's just a matter of preference and not an absolute. The user is better off determining which is better given their application. That is why both options exist.

@teohhanhui

This comment has been minimized.

teohhanhui commented Nov 27, 2018

From the manual:

‘GRUB_CMDLINE_LINUX’

Command-line arguments to add to menu entries for the Linux kernel.

‘GRUB_CMDLINE_LINUX_DEFAULT’

Unless ‘GRUB_DISABLE_RECOVERY’ is set to ‘true’, two menu entries will be generated for each Linux kernel: one default entry and one entry for recovery mode. This option lists command-line arguments to add only to the default menu entry, after those listed in ‘GRUB_CMDLINE_LINUX’.

https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html

@ejprice

This comment has been minimized.

ejprice commented Nov 27, 2018

Yes, the documentation is an excellent place to start when determining the correct usage as it applies to the problem domain that you are trying to solve.

However, if you're trying to use it to prove you are correct, then you've failed horribly. Mainly, because you are assuming you know what is best and not considering the application which the user is dealing with.

And, more importantly, you are missing the true joy of Unix and Unix/Linux like operating systems - there is more than one way to do things. It is up to the user to decide what is best given their requirements. To say there is only one correct way to do just about anything is categorically false. If it were true, we wouldn't have hundreds of varieties of GNU/Linux or the many flavors of *BSD Unix. In short, the software ecosystem that exists today wouldn't exist.

That said, this issue is about Docker not playing nicely (yet) with the new metacopy feature in Linux 4.19 and not about which is the proper method in which to append arguments to the Linux kernel at boot time. So, I'm signing off and not replying to further posts on this topic unless it relates to metacopy and Docker.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Nov 29, 2018

@kolyshkin

This comment has been minimized.

kolyshkin commented Nov 29, 2018

A workaround would be the same as in moby/moby#37993: add metacopy=off on a per-mount basis, if a parameter is available.

kolyshkin added a commit to kolyshkin/moby that referenced this issue Nov 29, 2018

overlay2: fix for Linux 4.19
Linux Kernel v.4.19 (kernel commit d5791044d2e574) introduces a new
feature called metadata copy up (or metacopy). Quoting kernel docs:

> When metadata only copy up feature is enabled, overlayfs will only copy
> up metadata (as opposed to whole file), when a metadata specific operation
> like chown/chmod is performed. Full file will be copied up later when
> file is opened for WRITE operation.

Unfortunately for Docker, this new feature is on by default (if the
feature is enabled at all by having CONFIG_OVERLAY_FS_METACOPY=y),
and it is not working with the naive diff driver; docker build fails
with something like

> dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

To fix, use the same mechanism as in commit 8422d85
("overlay2: use index=off if possible"), i.e. check if the
kernel module parameter is present, and if it is, add
`metacopy=off` to mount options.

Fixes docker/for-linux#480

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/moby that referenced this issue Nov 29, 2018

overlay2: fix for Linux 4.19 (metacopy=off)
Linux Kernel v.4.19 introduces a new overlay feature called metadata
copy up (or just metacopy). Quoting kernel docs:

> When metadata only copy up feature is enabled, overlayfs will only copy
> up metadata (as opposed to whole file), when a metadata specific operation
> like chown/chmod is performed. Full file will be copied up later when
> file is opened for WRITE operation.

Until the kernel commit d5791044d2e574 ("ovl: Provide a mount option
metacopy=on/off...") this feature was disabled by default.

Unfortunately for Docker, since the above commit the feature is on
by default in case CONFIG_OVERLAY_FS_METACOPY=y is set (which is up
to a distro vendor), and it is not working with the naive diff driver:
docker build leads to errors like this:

> dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

To fix, use the same mechanism as in commit 8422d85
("overlay2: use index=off if possible"), i.e. check if the
kernel module parameter is present, and if it is, add
`metacopy=off` to mount options.

Fixes docker/for-linux#480

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
@kolyshkin

This comment has been minimized.

kolyshkin commented Nov 29, 2018

Proposed fix: moby/moby#38296

thaJeztah added a commit to thaJeztah/docker that referenced this issue Nov 29, 2018

overlay2: fix for Linux 4.19 (metacopy=off)
Linux Kernel v.4.19 introduces a new overlay feature called metadata
copy up (or just metacopy). Quoting kernel docs:

> When metadata only copy up feature is enabled, overlayfs will only copy
> up metadata (as opposed to whole file), when a metadata specific operation
> like chown/chmod is performed. Full file will be copied up later when
> file is opened for WRITE operation.

Until the kernel commit d5791044d2e574 ("ovl: Provide a mount option
metacopy=on/off...") this feature was disabled by default.

Unfortunately for Docker, since the above commit the feature is on
by default in case CONFIG_OVERLAY_FS_METACOPY=y is set (which is up
to a distro vendor), and it is not working with the naive diff driver:
docker build leads to errors like this:

> dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

To fix, use the same mechanism as in commit 8422d85
("overlay2: use index=off if possible"), i.e. check if the
kernel module parameter is present, and if it is, add
`metacopy=off` to mount options.

Fixes docker/for-linux#480

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit a896b54)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@NamPNQ

This comment has been minimized.

NamPNQ commented Nov 30, 2018

Workaround for waiting next docker release
$ echo N | sudo tee /sys/module/overlay/parameters/metacopy
stop running container -> start container (restart does not work correctly)
ensure docker mount overlayfs not use metacopy=on
$ mount | grep "docker"

@rtizzy

This comment has been minimized.

rtizzy commented Dec 12, 2018

As other mentioned, this occurs on later versions of 4.19

Linux faithful 4.19.6-1-MANJARO

The mentioned echo N | sudo tee /sys/module/overlay/parameters/metacopy does resolve the issue.

@saro

This comment has been minimized.

saro commented Dec 14, 2018

Issue is still present in Archlinux kernel 4.19.8-arch1-1-ARCH

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment