Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash on docker run #19833

Closed
ei-grad opened this issue Jan 29, 2016 · 11 comments
Closed

Crash on docker run #19833

ei-grad opened this issue Jan 29, 2016 · 11 comments

Comments

@ei-grad
Copy link

ei-grad commented Jan 29, 2016

Update: I'm investiaging the issue now, it looks like it is problem on my machine...

Docker from latest archlinux package:

→ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.3
 Git commit:   a34a1d5-dirty
 Built:        Sat Jan 16 15:43:56 UTC 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.3
 Git commit:   a34a1d5-dirty
 Built:        Sat Jan 16 15:43:56 UTC 2016
 OS/Arch:      linux/amd64

→ docker info
Containers: 27
Images: 361
Server Version: 1.9.1
Storage Driver: btrfs
 Build Version: Btrfs v4.3.1
 Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.3.3-3-ARCH
Operating System: Arch Linux (containerized)
CPUs: 8
Total Memory: 31.12 GiB
Name: ei-grad
ID: 4CEN:IEHS:Z2U6:4Y2D:OPVA:PRHR:2DPJ:CL47:EAU2:Q6DD:C5GH:PPHZ

Stack trace:

$ docker build -t XXX/XXX:XXX .
Sending build context to Docker daemon 145.4 kBSending build context to Docker daemon 145.4 kB
Error response from daemon: Untar re-exec error: exit status 2: output: runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f62a169f5f8 m=3

goroutine 0 [idle]:

goroutine 19 [running]:
runtime.systemstack_switch()
    /usr/lib/go/src/runtime/asm_amd64.s:216 fp=0xc820024498 sp=0xc820024490
runtime.gc(0x0)
    /usr/lib/go/src/runtime/mgc.go:1006 +0x1db fp=0xc820024790 sp=0xc820024498
runtime.backgroundgc()
    /usr/lib/go/src/runtime/mgc.go:897 +0x3d fp=0xc8200247c0 sp=0xc820024790
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc8200247c8 sp=0xc8200247c0
created by runtime.startGC
    /usr/lib/go/src/runtime/mgc.go:870 +0x186

goroutine 1 [sleep, locked to thread]:
time.Sleep(0x186a0)
    /usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/docker/libnetwork/ipamutils.initGranularPredefinedNetworks(0x0, 0x0, 0x0)
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:38 +0x111
github.com/docker/libnetwork/ipamutils.init.1()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:17 +0x4d
github.com/docker/libnetwork/ipamutils.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils_linux.go:74 +0x59
github.com/docker/libnetwork/ipam.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipam/utils.go:81 +0x5e
github.com/docker/libnetwork/ipams/builtin.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipams/builtin/builtin.go:35 +0x45
github.com/docker/libnetwork.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/store.go:413 +0x9e
github.com/docker/docker/daemon.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/daemon/wait.go:17 +0x11e
github.com/docker/docker/api/server/router/local.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/router/local/volume.go:66 +0x85
github.com/docker/docker/api/server.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/server_unix.go:123 +0x94
main.init()
    /build/docker/src/docker/docker/flags.go:30 +0x8d

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x37

rax    0x0
rbx    0x7f62a1a0b720
rcx    0x7f62a169f5f8
rdx    0x6
rdi    0x28d
rsi    0x28f
rbp    0x174965f
rsp    0x7f629f8b5948
r8     0x7f62a1a0c790
r9     0x7f629f8b6700
r10    0x8
r11    0x206
r12    0x7f62980008c0
r13    0x170182c
r14    0x0
r15    0x8
rip    0x7f62a169f5f8
rflags 0x206
cs     0x33
fs     0x0
gs     0x0

ERROR: Build failed with: exit status 1
@GordonTheTurtle
Copy link

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

For more information about reporting issues, see CONTRIBUTING.md.

You don't have to include this information if this is a feature request

(This is an automated, informational response)

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

@ei-grad
Copy link
Author

ei-grad commented Jan 29, 2016

There is a strace output, if it could be helpful:

docker-build.txt

@ei-grad
Copy link
Author

ei-grad commented Jan 29, 2016

Oh, looks like it crashes on docker run:

→ docker run -it --rm busybox:latest                                                                               1 ↵
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
583635769552: Extracting [=======>                                           ]  98.3 kB/675.8 kB
b175bcb79023: Download complete 
Pulling repository docker.io/library/busybox
964092b7f3e5: Error pulling image (latest) from docker.io/library/busybox, endpoint: https://registry-1.docker.io/v1/, ApplyLayer exit status 2 stdout:  stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f0c76c205f8 m=0

goroutine 0 [idle]:

goroutine 1 [runnable, locked to thread]:
runtime.Gosched()
    /usr/lib/go/src/runtime/proc.go:166 +0x14 fp=0xc820145c80 sp=0xc820145c70
runtime.mallocgc(0x10, 0x1005e80, 0x1, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:725 +0x58a fp=0xc820145d50 sp=0xc820145c80
runtime.newobject(0x1005e80, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:763 +0x42 fp=0xc820145d78 sp=0xc820145d50
github.com/docker/libnetwork/ipamutils.initGranularPredefinedNetworks(0x0, 0x0, 0x0)
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:38 +0x12b fp=0xc820145e20 sp=0xc820145d78
github.com/docker/libnetwork/ipamutils.init.1()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:17 +0x4d fp=0xc820145e40 sp=0xc820145e20
github.com/docker/libnetwork/ipamutils.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils_linux.go:74 +0x59 fp=0xc820145e48 sp=0xc820145e40
github.com/docker/libnetwork/ipam.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipam/utils.go:81 +0x5e fp=0xc820145e50 sp=0xc820145e48
github.com/docker/libnetwork/ipams/builtin.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipams/builtin/builtin.go:35 +0x45 fp=0xc820145e58 sp=0xc820145e50
github.com/docker/libnetwork.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/store.go:413 +0x9e fp=0xc820145e60 sp=0xc820145e58
github.com/docker/docker/daemon.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/daemon/wait.go:17 +0x11e fp=0xc820145e88 sp=0xc820145e60
github.com/docker/docker/api/server/router/local.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/router/local/volume.go:66 +0x85 fp=0xc820145e90 sp=0xc820145e88
github.com/docker/docker/api/server.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/server_unix.go:123 +0x94 fp=0xc820145e98 sp=0xc820145e90
main.init()
    /build/docker/src/docker/docker/flags.go:30 +0x8d fp=0xc820145f00 sp=0xc820145e98
runtime.main()
    /usr/lib/go/src/runtime/proc.go:100 +0x27f fp=0xc820145f50 sp=0xc820145f00
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc820145f58 sp=0xc820145f50

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x37

rax    0x0
rbx    0x7f0c76f8c720
rcx    0x7f0c76c205f8
rdx    0x6
rdi    0xf30
rsi    0xf30
rbp    0x174965f
rsp    0x7ffe5aa520b8
r8     0x7f0c76f8d790
r9     0x7f0c777427c0
r10    0x8
r11    0x206
r12    0x30db040
r13    0x170182c
r14    0x0
r15    0x8
rip    0x7f0c76c205f8
rflags 0x206
cs     0x33
964092b7f3e5: Error pulling image (latest) from docker.io/library/busybox, ApplyLayer exit status 2 stdout:  stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f0c76c205f8 m=0

goroutine 0 [idle]:

goroutine 1 [runnable, locked to thread]:
runtime.Gosched()
    /usr/lib/go/src/runtime/proc.go:166 +0x14 fp=0xc820145c80 sp=0xc820145c70
runtime.mallocgc(0x10, 0x1005e80, 0x1, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:725 +0x58a fp=0xc820145d50 sp=0xc820145c80
runtime.newobject(0x1005e80, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:763 +0x42 fp=0xc820145d78 sp=0xc820145d50
github.com/docker/libnetwork/ipamutils.initGranularPredefinedNetworks(0x0, 0x0, 0x0)
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:38 +0x12b fp=0xc820145e20 sp=0xc820145d78
github.com/docker/libnetwork/ipamutils.init.1()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:17 +0x4d fp=0xc820145e40 sp=0xc820145e20
github.com/docker/libnetwork/ipamutils.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils_linux.go:74 +0x59 fp=0xc820145e48 sp=0xc820145e40
github.com/docker/libnetwork/ipam.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipam/utils.go:81 +0x5e fp=0xc820145e50 sp=0xc820145e48
github.com/docker/libnetwork/ipams/builtin.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipams/builtin/builtin.go:35 +0x45 fp=0xc820145e58 sp=0xc820145e50
github.com/docker/libnetwork.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/store.go:413 +0x9e fp=0xc820145e60 sp=0xc820145e58
github.com/docker/docker/daemon.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/daemon/wait.go:17 +0x11e fp=0xc820145e88 sp=0xc820145e60
github.com/docker/docker/api/server/router/local.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/router/local/volume.go:66 +0x85 fp=0xc820145e90 sp=0xc820145e88
github.com/docker/docker/api/server.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/server_unix.go:123 +0x94 fp=0xc820145e98 sp=0xc820145e90
main.init()
    /build/docker/src/docker/docker/flags.go:30 +0x8d fp=0xc820145f00 sp=0xc820145e98
runtime.main()
    /usr/lib/go/src/runtime/proc.go:100 +0x27f fp=0xc820145f50 sp=0xc820145f00
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc820145f58 sp=0xc820145f50

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x37

rax    0x0
rbx    0x7f0c76f8c720
rcx    0x7f0c76c205f8
rdx    0x6
rdi    0xf30
rsi    0xf30
rbp    0x174965f
rsp    0x7ffe5aa520b8
r8     0x7f0c76f8d790
r9     0x7f0c777427c0
r10    0x8
r11    0x206
r12    0x30db040
r13    0x170182c
r14    0x0
r15    0x8
rip    0x7f0c76c205f8
rflags 0x206
cs     0x33
fs     0x0
gs     0x0
Error pulling image (latest) from docker.io/library/busybox, ApplyLayer exit status 2 stdout:  stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f0c76c205f8 m=0

goroutine 0 [idle]:

goroutine 1 [runnable, locked to thread]:
runtime.Gosched()
    /usr/lib/go/src/runtime/proc.go:166 +0x14 fp=0xc820145c80 sp=0xc820145c70
runtime.mallocgc(0x10, 0x1005e80, 0x1, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:725 +0x58a fp=0xc820145d50 sp=0xc820145c80
runtime.newobject(0x1005e80, 0xc820525f80)
    /usr/lib/go/src/runtime/malloc.go:763 +0x42 fp=0xc820145d78 sp=0xc820145d50
github.com/docker/libnetwork/ipamutils.initGranularPredefinedNetworks(0x0, 0x0, 0x0)
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:38 +0x12b fp=0xc820145e20 sp=0xc820145d78
github.com/docker/libnetwork/ipamutils.init.1()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils.go:17 +0x4d fp=0xc820145e40 sp=0xc820145e20
github.com/docker/libnetwork/ipamutils.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipamutils/utils_linux.go:74 +0x59 fp=0xc820145e48 sp=0xc820145e40
github.com/docker/libnetwork/ipam.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipam/utils.go:81 +0x5e fp=0xc820145e50 sp=0xc820145e48
github.com/docker/libnetwork/ipams/builtin.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/ipams/builtin/builtin.go:35 +0x45 fp=0xc820145e58 sp=0xc820145e50
github.com/docker/libnetwork.init()
    /build/docker/src/docker/vendor/src/github.com/docker/libnetwork/store.go:413 +0x9e fp=0xc820145e60 sp=0xc820145e58
github.com/docker/docker/daemon.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/daemon/wait.go:17 +0x11e fp=0xc820145e88 sp=0xc820145e60
github.com/docker/docker/api/server/router/local.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/router/local/volume.go:66 +0x85 fp=0xc820145e90 sp=0xc820145e88
github.com/docker/docker/api/server.init()
    /build/docker/src/docker/.gopath/src/github.com/docker/docker/api/server/server_unix.go:123 +0x94 fp=0xc820145e98 sp=0xc820145e90
main.init()
    /build/docker/src/docker/docker/flags.go:30 +0x8d fp=0xc820145f00 sp=0xc820145e98
runtime.main()
    /usr/lib/go/src/runtime/proc.go:100 +0x27f fp=0xc820145f50 sp=0xc820145f00
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc820145f58 sp=0xc820145f50

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x37

rax    0x0
rbx    0x7f0c76f8c720
rcx    0x7f0c76c205f8
rdx    0x6
rdi    0xf30
rsi    0xf30
rbp    0x174965f
rsp    0x7ffe5aa520b8
r8     0x7f0c76f8d790
r9     0x7f0c777427c0
r10    0x8
r11    0x206
r12    0x30db040
r13    0x170182c
r14    0x0
r15    0x8
rip    0x7f0c76c205f8
rflags 0x206
cs     0x33
fs     0x0
gs     0x0

@ei-grad ei-grad changed the title Crash on docker build Crash on docker run Jan 29, 2016
@cpuguy83
Copy link
Member

pthread_create failed: Resource temporarily unavailable

@ei-grad
Copy link
Author

ei-grad commented Jan 29, 2016

Other programs on my machine worked ok. Don't know what could cause such issue, but it is not the first time I faced with thread/process running failure in containers started by a long-running docker service.

@ei-grad ei-grad closed this as completed Jan 29, 2016
@devenv
Copy link

devenv commented Feb 25, 2016

Why was it closed? I'm having exactly the same issue...

@icetan
Copy link

icetan commented Feb 26, 2016

I just suddenly started getting this error when pulling images. I haven't upgraded docker or anything else in my environment and everything worked fine up until today.

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Fri Feb 12 00:52:02 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Fri Feb 12 00:52:02 2016
 OS/Arch:      linux/amd64

@devenv
Copy link

devenv commented Feb 27, 2016

I'm getting this with docker 1.9.1 on openSUSE Tumbleweed with kernel 4.4.2, the same happens with 1.10 and 1.11
I'll try Leap later with much older kernel as it seems related...

@devenv
Copy link

devenv commented Feb 28, 2016

Downgrading to kernel 4.1 did the trick...

@icetan
Copy link

icetan commented Feb 29, 2016

I'm on Manjaro with kernel 4.4.2 which is essentially Arch.

Found this issue #19124 which points to this: https://bugs.archlinux.org/task/47303

The Arch issue describes some of the symptoms I get.

I've set TasksMax=infinity in my docker.service and I'm trying that out now, not had any problems yet.

@devenv
Copy link

devenv commented Mar 1, 2016

TasksMax=infinity didn't help in my case...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ei-grad @devenv @cpuguy83 @icetan @GordonTheTurtle