docker 19.03-RC3: cp broken with debian containers (arm?)

[dubo-dubon-duponey]

Description

docker cp is broken with Debian containers (on armhf).

Steps to reproduce the issue:

1. install the latest docker 19.03 on armhf
2. docker run --name foo -d debian:buster-slim sleep 1000
3. docker cp foo:/root/.profile .

Describe the results you received:

Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/arm-linux-gnueabihf/libnss_files.so.2: symbol __libc_readline_unlocked, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference : exit status 127

Describe the results you expected:

Work.

Additional information you deem important (e.g. issue happens only occasionally):

This works fine with:

• Alpine (musl)
• BusyBox

This also works fine with Debian on D4M using 19.03-RC2

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.0-rc3
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        27fcb77
 Built:             Thu Jun 20 02:14:50 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-rc3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       27fcb77
  Built:            Thu Jun 20 02:08:53 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 10
  Running: 7
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 19.03.0-rc3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.42-v7+
 Operating System: Raspbian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 926.1MiB
 Name: raspberrypi
 ID: SJG3:SBJ2:7E63:4KEV:SJNJ:6HXM:UX2S:XOOT:MN6D:2WTM:NBBT:FNIP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Additional environment details (AWS, VirtualBox, physical, etc.):

raspberry pi

Either this is limited to armhf, or this is only in RC3 and not in RC2.

@tiborvass also hinted at:

• [19.03] Add chroot back to Tar/Untar without the previously introduced regression docker/engine#280
• [19.03] Revert Pass root to chroot to for chroot Tar/Untar (CVE-2018-15664)  docker/engine#275

[tiborvass]

@dubo-dubon-duponey can you try with MOBY_DISABLE_PIGZ=1 when starting the daemon?

[dubo-dubon-duponey]

/etc/default/docker:

export MOBY_DISABLE_PIGZ=1

Restarted the daemon.

Same error.

[thaJeztah]

Either this is limited to armhf, or this is only in RC3 and not in RC2.

This is a bit confusing; Did you run RC2 on armhf, and the problem didn't show?

[dubo-dubon-duponey]

Problem is visible on a raspberry pi (armhf) with RC3.

Not visible on Docker4Mac with RC2.

[tiborvass]

@dubo-dubon-duponey if you could test with rc2 on armhf it could be a useful datapoint.

[dubo-dubon-duponey]

BUT BUT BUT THAT WOULD KILL MY RASPBERRY UPTIME!!!!

...

Ok dad, on it :-)

[dubo-dubon-duponey]

Broken:

5:19.03.0~2.3.rc3-0~raspbian-stretch
5:19.03.0~2.2.rc2-0~raspbian-stretch

Working:

5:19.03.0~1.5.beta5-0~raspbian-stretch
5:19.03.0~1.1.beta1-0~raspbian-stretch

No clue why it would work on amd64/mac with RC2.

Not sure if the version of containerd would be relevant or not.

[thaJeztah]

Possibly relevant discussion https://lists.debian.org/debian-glibc/2016/03/msg00154.html

(Also https://stackoverflow.com/questions/56590562/debian-9-libm-and-symbol-get-cpu-features-version-glibc-private-not-defined)

[guss77]

Happens to me with "stable" 5:19.03.0~3-0~ubuntu-bionic (from the bionic/stable repository) on Ubuntu bionic (18.04.2):

# docker cp container:/somefile somefile
Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
: exit status 127

Reverting to 5:18.09.8~3-0~ubuntu-bionic solved the problem.

From https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911090 it sounds like the gcc for building 19.03 has changed in a way that is incompatible with the gcc that was used to build the glibc from stable bionic.

[andrewhsu]

Hmm...I think we need a closer look at this. I see this on x86_64 as well with docker-ce 19.03.0. cc @tiborvass

[canassa]

I started having this issue about 12 hours ago with the python:3.6 image. It crashes during the docker cp command

[hhromic]

Same here, just got the following error while using a debian/buster image:

Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
: exit status 127

EDIT: sorry pressed "send" before finishing.

I'm using a stable Debian Stretch system as host in an amd64 arch machine.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        aeac9490dc
 Built:             Wed Jul 17 18:14:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       aeac9490dc
  Built:            Wed Jul 17 18:12:33 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[tiborvass]

Thanks for all your help, I'll take this one. Will report back.

[docwhat]

I think this is actually a kind of statically compiled (CGO_ENABLED=0) Go problem.

I think go is loading up the NSS libraries in attempt to behave like any other OS command.

I know that for systems where NSS doesn't exist that Go has go-only versions built in.

[imxiny]

Same error , I need help😭

[thaJeztah]

Yes, I think the cause was identified; we're looking at the right fix (there's a couple of options being discussed)

[justincormack]

Fixed in #39612

[hhromic]

@justincormack can confirm my original issue is resolved now after the update.
Thanks for the very fast response and solution. Cheers!

[thaJeztah]

For future reference; a CVE was assigned to this issue: CVE-2019-14271

[vanti]

I am seeing same error with 'docker cp' on 19.03.5 in Ubuntu 18.04.

[belek]

Same error on 19.03.5 Debian GNU/Linux 9.11 (stretch)

[half2me]

I'm having the same issue on stretch with kernel 4.9.0-6-amd64 and Docker version 19.03.5, build 633a0ea838

[Daky]

for me only the machine running
Docker version 19.03.1, build 74b1e89
won't have this issue
I have try to use
Docker version 19.03.1, build 74b1e89e8a
it will still fail with the same error

[GarbageYard]

I am also getting the same error on CentOS Linux release 7.6.1810 (Core)

Docker Engine - Community (Version: 19.03.0)

[jdevuyst-pica9]

I am seeing the error with the following:

devuyst@jdevuyst-Inspiron-5579:~$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
jdevuyst@jdevuyst-Inspiron-5579:~$ 
jdevuyst@jdevuyst-Inspiron-5579:~$ 
jdevuyst@jdevuyst-Inspiron-5579:~$ apt list --installed | grep docker 

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

docker-ce/xenial,now 5:19.03.6~3-0~ubuntu-xenial amd64 [installed]
docker-ce-cli/xenial,now 5:19.03.6~3-0~ubuntu-xenial amd64 [installed,automatic]
jdevuyst@jdevuyst-Inspiron-5579:~$

[itchyny]

For people who is lead to this page after searching with the error message, please upgrade the version of Docker >= 19.03.8.

[half2me]

Upgrading to Docker version 19.03.8 fixed this for me