Panic when pulling an image, ApplyLayer - too many links" #16755

Soulou · 2015-10-04T21:36:03Z

Hello everyone,

I wanted to pull nsqio/nsq:v0.3.6 but my docker daemon crashed (with all my containers :() with the following error.

It only occured on one server (docker 1.7.1, kernel 3.13.x, btrfs backend, lot of free ram and lot of free disk space), everything went well on the others. The error is repeatable, I have not restarted the server yet.

I'm wondering the cause of this error, and maybe a fix for handling this error could be done to avoid a panic.

time="2015-10-04T23:19:55.905551060+02:00" level=info msg="POST /v1.16/images/create?fromImage=nsqio%2Fnsq&tag=v0.3.6"
time="2015-10-04T23:19:58.897438752+02:00" level=error msg="Error from V2 registry: ApplyLayer exit status 1 stdout:  stderr: link /bin/[ /bin/run-parts: too many links"  
panic: runtime error: invalid memory address or nil pointer dereference 
[signal 0xb code=0x1 addr=0x20 pc=0x62fc6a] 

goroutine 131472 [running]: 
bufio.(*Writer).flush(0xc20c46a9c0, 0x0, 0x0) 
        /usr/local/go/src/bufio/bufio.go:530 +0xda 
bufio.(*Writer).Flush(0xc20c46a9c0, 0x0, 0x0) 
        /usr/local/go/src/bufio/bufio.go:519 +0x3a 
net/http.(*response).Flush(0xc208fe74a0) 
        /usr/local/go/src/net/http/server.go:1047 +0x4c 
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc209bc4600, 0xc2096ba140, 0xbd, 0x137, 0xbd, 0x0, 0x0) 
        /go/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:21 +0x145 
github.com/docker/docker/pkg/progressreader.(*Config).Read(0xc208a77030, 0xc2094cc000, 0x8000, 0x8000, 0x4000, 0x0, 0x0) 
        /go/src/github.com/docker/docker/pkg/progressreader/progressreader.go:37 +0x2e5 
io.Copy(0x7f622cf1d2a8, 0xc20b24b1b0, 0x7f6228421000, 0xc208a77030, 0xbfe7e8, 0x0, 0x0) 
        /usr/local/go/src/io/io.go:362 +0x1f6 
github.com/docker/docker/graph.func·008(0xc209892140, 0x0, 0x0) 
        /go/src/github.com/docker/docker/graph/pull.go:602 +0xd42 
github.com/docker/docker/graph.func·009(0xc209892140) 
        /go/src/github.com/docker/docker/graph/pull.go:626 +0x2f 
created by github.com/docker/docker/graph.(*TagStore).pullV2Tag 
        /go/src/github.com/docker/docker/graph/pull.go:627 +0x2671

docker version

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info

Containers: 62
Images: 384
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-65-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 15.59 GiB
Name: <hostname>
ID: LDCP:QPPH:XD7G:R2XJ:TWW3:ALIX:EA5Q:DO5M:RVCI:VZBE:HVP3:A3R2
Registry: https://index.docker.io/v1/

The text was updated successfully, but these errors were encountered:

j16sdiz · 2015-10-07T05:18:15Z

https://bugzilla.kernel.org/show_bug.cgi?id=15762#c11

Soulou · 2015-10-07T13:10:27Z

@j16sdiz Thank you for the link

$ stop docker
$ umount /var/lib/docker
$ btrfstune -r <device>
$ mount /var/lib/docker
$ start docker
$ docker pull nsqio/nsq:v0.3.6

Results is identical:

v0.3.6: Pulling from nsqio/nsq
cfa753dfea5e: Extracting 666.8 kB/666.8 kB
cfa753dfea5e: Error downloading dependent layers 
2c5abc25ceae: Downloading 3.416 MB/26.04 MB
2c5abc25ceae: Download complete 
f13605ba9cdb: Download complete 
21af2da842be: Download complete 
3cd00aaa505d: Download complete 
3cd00aaa505d: Error pulling image (v0.3.6) from nsqio/nsq, endpoint: https://reg3cd00aaa505d: Error pulling image (v0.3.6) from nsqio/nsq, ApplyLayer exit statuError pulling image (v0.3.6) from nsqio/nsq, ApplyLayer exit status 1 stdout:  stderr: link /bin/[ /bin/run-parts: too many links

j16sdiz · 2015-10-07T17:37:06Z

@Soulou I know this sounds silly.... But do you have >65535 images / containers (include stopped / dead container)?

Soulou · 2015-10-07T20:34:26Z

No question sounds silly, I'm glad you try helping. No, as stated in my docker info (first comment), there are only around 400 images and 60 containers

j16sdiz · 2015-10-08T11:56:11Z

Are all your images busybox based?
busybox have higher then usual number of hardlinks.

Btrfs limit is 64k for extref, 255 without extref.

Soulou · 2015-10-08T13:58:33Z

Nop, my images are all based on ubuntu:14.04.

I find this error weird, event after deleting some images, it doesn't change anything.

tonistiigi · 2015-10-14T18:11:02Z

I think the panic part of the bug is fixed in 1.8, not sure what causes the "too many links" though.

coyotwill · 2015-10-27T22:21:37Z

@j16sdiz : I'm seeing the same error on a busybox-based image.
FATA[0000] Error pulling image (0.10.0) from myregistry:5000/mystuff, ApplyLayer exit status 1 stdout: stderr: link /bin/[ /bin/route: too many links

You seems to understand what the problem may be, could you please elaborate ?

How can I check the number of hard-links or soft-links ? And do they add up to other images ?
I'm asking because pulling this image used to work .. until today. And now I can't even pull old images that used to work and are already there in the local registry.

[EDIT] For people ending here after a google search: Changing the base image from busybox to something else solved my issue (in my case I changed to python:3.5).
But I'm still interested in understanding the issue. And especially how the number of links adds up between the different images.

Thanks

j16sdiz · 2015-10-29T18:24:48Z

@coyotwill busybox have (almost) all files in /sbin /bin and /usr/bin hardlinked to the same binary.
you can check the number of hardlink with stat(1) command.

$ stat .
  File: ‘.’
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fc00h/64512d    Inode: 59779105    Links: 3
Access: (0755/drwxr-xr-x)  Uid: ( 1000/    sdiz)   Gid: ( 1000/    sdiz)
Access: 2015-10-30 02:15:17.194821805 +0800
Modify: 2015-10-30 02:15:37.754654694 +0800
Change: 2015-10-30 02:15:37.754654694 +0800
 Birth: -

coyotwill · 2015-10-29T20:23:40Z

Thanks for your answer @j16sdiz
But I'm still wondering how this limit is calculated. Because I was able to pull and run the exact same image a few days ago. And now I can't even pull it anymore.
I also tried to remove old versions of the image from the local registry and it didn't help either.

LK4D4 · 2016-09-16T18:04:56Z

Flush panic was fixed. Too many links probably too, wasn't reported since. Feel free to open new issue if problem persists.

FlorentATo · 2017-01-16T06:40:20Z

Hello,
I do have the same issue, regardless of what flavor of busybox it try to run (musl, glibc, uclibc):

Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
4b0bc1c4050b: Extracting [==================================================>] 677.6 kB/677.6 kB
docker: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: link /bin/[ /bin/reset: too many links.
See 'docker run --help'.

Docker version:

➜  ~ sudo docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:17:57 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:17:57 2017
 OS/Arch:      linux/amd64

Docker info:

➜  ~ sudo docker info   
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 54
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v3.17
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.14.32-xxxx-grs-ipv6-64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.834 GiB
Name: kimsufi-ato
ID: LJYT:42YF:XSH5:EDKY:YAMG:RT2Z:KBVJ:B52M:ZHWW:XZF2:G6ED:PDOO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical server, Debian Jessie 8.7. All freshly upgraded.

tonistiigi · 2017-01-17T21:29:45Z

Btrfs used to have a very low hardlink limit if files existed in same directories like they do for busybox. This was fixed in Linux and should be default since 3.12. man mkfs.btrfs:

       extref
           (default since btrfs-progs 3.12, kernel support since 3.7)

           increased hardlink limit per file in a directory to 65536, older
           kernels supported a varying number of hardlinks depending on the
           sum of all file name sizes that can be stored into one metadata
           block

For existing devices you can use btrfstune -r

thaJeztah · 2017-01-17T23:17:34Z

@tonistiigi should that be added to the user guide? https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/

tonistiigi · 2017-01-17T23:20:42Z

@thaJeztah Maybe. This issue should be old enough to not appear. I'm sure there are worse btrfs issues than this in old kernels. But if @FlorentATo confirms that this helped, then we could add it for sure.

thaJeztah added the area/storage/btrfs label Oct 7, 2015

LK4D4 closed this as completed Sep 16, 2016

thaJeztah added the kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. label Sep 17, 2016

sedlund mentioned this issue Aug 21, 2018

kubeadm init failed to pull image kubernetes/kubernetes#67492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Panic when pulling an image, ApplyLayer - too many links" #16755

Panic when pulling an image, ApplyLayer - too many links" #16755

Soulou commented Oct 4, 2015

j16sdiz commented Oct 7, 2015

Soulou commented Oct 7, 2015

j16sdiz commented Oct 7, 2015

Soulou commented Oct 7, 2015

j16sdiz commented Oct 8, 2015

Soulou commented Oct 8, 2015

tonistiigi commented Oct 14, 2015

coyotwill commented Oct 27, 2015

j16sdiz commented Oct 29, 2015

coyotwill commented Oct 29, 2015

LK4D4 commented Sep 16, 2016

FlorentATo commented Jan 16, 2017

tonistiigi commented Jan 17, 2017

thaJeztah commented Jan 17, 2017

tonistiigi commented Jan 17, 2017

Panic when pulling an image, ApplyLayer - too many links" #16755

Panic when pulling an image, ApplyLayer - too many links" #16755

Comments

Soulou commented Oct 4, 2015

j16sdiz commented Oct 7, 2015

Soulou commented Oct 7, 2015

j16sdiz commented Oct 7, 2015

Soulou commented Oct 7, 2015

j16sdiz commented Oct 8, 2015

Soulou commented Oct 8, 2015

tonistiigi commented Oct 14, 2015

coyotwill commented Oct 27, 2015

j16sdiz commented Oct 29, 2015

coyotwill commented Oct 29, 2015

LK4D4 commented Sep 16, 2016

FlorentATo commented Jan 16, 2017

tonistiigi commented Jan 17, 2017

thaJeztah commented Jan 17, 2017

tonistiigi commented Jan 17, 2017