Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker 19.03-RC3: cp broken with debian containers (arm?) #39449

Closed
dubo-dubon-duponey opened this issue Jul 1, 2019 · 19 comments
Assignees

Comments

@dubo-dubon-duponey
Copy link

@dubo-dubon-duponey dubo-dubon-duponey commented Jul 1, 2019

Description

docker cp is broken with Debian containers (on armhf).

Steps to reproduce the issue:

  1. install the latest docker 19.03 on armhf
  2. docker run --name foo -d debian:buster-slim sleep 1000
  3. docker cp foo:/root/.profile .

Describe the results you received:

Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/arm-linux-gnueabihf/libnss_files.so.2: symbol __libc_readline_unlocked, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference : exit status 127

Describe the results you expected:

Work.

Additional information you deem important (e.g. issue happens only occasionally):

This works fine with:

  • Alpine (musl)
  • BusyBox

This also works fine with Debian on D4M using 19.03-RC2

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.0-rc3
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        27fcb77
 Built:             Thu Jun 20 02:14:50 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-rc3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       27fcb77
  Built:            Thu Jun 20 02:08:53 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 10
  Running: 7
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 19.03.0-rc3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.42-v7+
 Operating System: Raspbian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 926.1MiB
 Name: raspberrypi
 ID: SJG3:SBJ2:7E63:4KEV:SJNJ:6HXM:UX2S:XOOT:MN6D:2WTM:NBBT:FNIP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

Additional environment details (AWS, VirtualBox, physical, etc.):

raspberry pi

Either this is limited to armhf, or this is only in RC3 and not in RC2.

@tiborvass also hinted at:

@dubo-dubon-duponey dubo-dubon-duponey changed the title docker cp broken with debian containers (arm) docker 19.03-RC3: cp broken with debian containers (arm) Jul 1, 2019
@dubo-dubon-duponey dubo-dubon-duponey changed the title docker 19.03-RC3: cp broken with debian containers (arm) docker 19.03-RC3: cp broken with debian containers (arm?) Jul 1, 2019
@tiborvass

This comment has been minimized.

Copy link
Collaborator

@tiborvass tiborvass commented Jul 2, 2019

@dubo-dubon-duponey can you try with MOBY_DISABLE_PIGZ=1 when starting the daemon?

@dubo-dubon-duponey

This comment has been minimized.

Copy link
Author

@dubo-dubon-duponey dubo-dubon-duponey commented Jul 2, 2019

/etc/default/docker:

export MOBY_DISABLE_PIGZ=1

Restarted the daemon.

Same error.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jul 2, 2019

Either this is limited to armhf, or this is only in RC3 and not in RC2.

This is a bit confusing; Did you run RC2 on armhf, and the problem didn't show?

@dubo-dubon-duponey

This comment has been minimized.

Copy link
Author

@dubo-dubon-duponey dubo-dubon-duponey commented Jul 2, 2019

Problem is visible on a raspberry pi (armhf) with RC3.

Not visible on Docker4Mac with RC2.

@tiborvass

This comment has been minimized.

Copy link
Collaborator

@tiborvass tiborvass commented Jul 2, 2019

@dubo-dubon-duponey if you could test with rc2 on armhf it could be a useful datapoint.

@dubo-dubon-duponey

This comment has been minimized.

Copy link
Author

@dubo-dubon-duponey dubo-dubon-duponey commented Jul 2, 2019

BUT BUT BUT THAT WOULD KILL MY RASPBERRY UPTIME!!!!

...

Ok dad, on it :-)

@dubo-dubon-duponey

This comment has been minimized.

Copy link
Author

@dubo-dubon-duponey dubo-dubon-duponey commented Jul 2, 2019

Broken:

5:19.03.0~2.3.rc3-0~raspbian-stretch
5:19.03.0~2.2.rc2-0~raspbian-stretch

Working:

5:19.03.0~1.5.beta5-0~raspbian-stretch
5:19.03.0~1.1.beta1-0~raspbian-stretch

No clue why it would work on amd64/mac with RC2.

Not sure if the version of containerd would be relevant or not.

@guss77

This comment has been minimized.

Copy link

@guss77 guss77 commented Jul 23, 2019

Happens to me with "stable" 5:19.03.0~3-0~ubuntu-bionic (from the bionic/stable repository) on Ubuntu bionic (18.04.2):

# docker cp container:/somefile somefile
Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
: exit status 127

Reverting to 5:18.09.8~3-0~ubuntu-bionic solved the problem.

From https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911090 it sounds like the gcc for building 19.03 has changed in a way that is incompatible with the gcc that was used to build the glibc from stable bionic.

@andrewhsu

This comment has been minimized.

Copy link
Contributor

@andrewhsu andrewhsu commented Jul 23, 2019

Hmm...I think we need a closer look at this. I see this on x86_64 as well with docker-ce 19.03.0. cc @tiborvass

@canassa

This comment has been minimized.

Copy link

@canassa canassa commented Jul 23, 2019

I started having this issue about 12 hours ago with the python:3.6 image. It crashes during the docker cp command

@hhromic

This comment has been minimized.

Copy link

@hhromic hhromic commented Jul 24, 2019

Same here, just got the following error while using a debian/buster image:

Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
: exit status 127

EDIT: sorry pressed "send" before finishing.

I'm using a stable Debian Stretch system as host in an amd64 arch machine.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        aeac9490dc
 Built:             Wed Jul 17 18:14:03 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       aeac9490dc
  Built:            Wed Jul 17 18:12:33 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
@tiborvass tiborvass self-assigned this Jul 24, 2019
@tiborvass

This comment has been minimized.

Copy link
Collaborator

@tiborvass tiborvass commented Jul 24, 2019

Thanks for all your help, I'll take this one. Will report back.

@docwhat

This comment has been minimized.

Copy link

@docwhat docwhat commented Jul 25, 2019

I think this is actually a kind of statically compiled (CGO_ENABLED=0) Go problem.

I think go is loading up the NSS libraries in attempt to behave like any other OS command.

I know that for systems where NSS doesn't exist that Go has go-only versions built in.

@imxiny

This comment has been minimized.

Copy link

@imxiny imxiny commented Jul 25, 2019

Same error , I need help😭

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jul 25, 2019

Yes, I think the cause was identified; we're looking at the right fix (there's a couple of options being discussed)

@justincormack

This comment has been minimized.

Copy link
Contributor

@justincormack justincormack commented Jul 26, 2019

Fixed in #39612

@hhromic

This comment has been minimized.

Copy link

@hhromic hhromic commented Jul 26, 2019

@justincormack can confirm my original issue is resolved now after the update.
Thanks for the very fast response and solution. Cheers!

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Sep 5, 2019

For future reference; a CVE was assigned to this issue: CVE-2019-14271

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.