docker daemon panic #17625

alexmac · 2015-11-02T20:10:04Z

I'm experiencing a number of panics of the docker daemon with a self-compiled build of the 1.9 branch. I've not yet got any repro steps but here are the details incase the stack traces are enough to spot the bug. This was built from git commit 80439a4 and running on an up to date Amazon linux AMI:

uname -a
Linux ip-172-31-29-231 4.1.10-17.31.amzn1.x86_64 #1 SMP Sat Oct 24 01:31:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client:
 Version:      1.9.0-dev
 API version:  1.22
 Go version:   go1.4.2
 Git commit:   80439a4
 Built:        Sun Oct 25 05:34:53 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-dev
 API version:  1.22
 Go version:   go1.4.2
 Git commit:   80439a4
 Built:        Sun Oct 25 05:34:53 UTC 2015
 OS/Arch:      linux/amd64

docker info:

docker info
Containers: 2
Images: 221
Server Version: 1.9.0-dev
Storage Driver: devicemapper
 Pool Name: docker-202:16-5767169-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 10.09 GB
 Data Space Total: 107.4 GB
 Data Space Available: 97.28 GB
 Metadata Space Used: 14.6 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.133 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.10-17.31.amzn1.x86_64
Operating System: Amazon Linux AMI 2015.09
CPUs: 2
Total Memory: 3.862 GiB
Name: ip-172-31-29-231
ID: BJRN:7OAY:G7XX:MOTV:P4VC:RLOA:T5GK:ZKOU:Q5GB:IJA6:42FN:3BI3

End of the docker log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x6e056a]

goroutine 34105 [running]:
bufio.(*Writer).flush(0xc20ae34280, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:530 +0xda
bufio.(*Writer).Flush(0xc20ae34280, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:519 +0x3a
net/http.(*response).Flush(0xc20af5ebe0)
    /usr/lib/golang/src/net/http/server.go:1047 +0x4c
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc2089f2630, 0xc20a3dc6e0, 0xa6, 0x155, 0xa6, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:22 +0x145
github.com/docker/docker/pkg/progressreader.updateProgress(0xc20a63e9a0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/progressreader/progressreader.go:67 +0x101
github.com/docker/docker/pkg/progressreader.(*Config).Read(0xc20a63e9a0, 0xc208d6c000, 0x8000, 0x8000, 0x8000, 0x7f0f2f175240, 0xc208032120)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/progressreader/progressreader.go:41 +0x229
io.Copy(0x7f0f2c4d59f0, 0xc209afbd90, 0x7f0f2c4e2980, 0xc20a63e9a0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
github.com/docker/docker/graph.func·016()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:270 +0xb9
created by github.com/docker/docker/graph.(*v2Pusher).pushV2Image
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:282 +0xa54

goroutine 1 [chan receive, 763 minutes]:
main.(*DaemonCli).CmdDaemon(0xc2084dd270, 0xc20800a020, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/docker/daemon.go:308 +0x20a7
main.handleGlobalDaemonFlag()
    /home/ec2-user/build/src/github.com/docker/docker/docker/daemon.go:59 +0xe6
main.main()
    /home/ec2-user/build/src/github.com/docker/docker/docker/docker.go:61 +0x2ac

goroutine 5 [syscall, 764 minutes]:
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x35

goroutine 17 [syscall, 764 minutes, locked to thread]:
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:2232 +0x1

goroutine 10 [chan receive, 763 minutes]:
github.com/docker/docker/api/server.(*Server).ServeAPI(0xc2080b2cc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:94 +0x1b6
main.func·007()
    /home/ec2-user/build/src/github.com/docker/docker/docker/daemon.go:255 +0x3b
created by main.(*DaemonCli).CmdDaemon
    /home/ec2-user/build/src/github.com/docker/docker/docker/daemon.go:261 +0x1571

goroutine 13 [IO wait]:
net.(*pollDesc).Wait(0xc2084de450, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2084de450, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2084de3f0, 0x0, 0x7f0f2f179500, 0xc20ac3bdc8)
    /usr/lib/golang/src/net/fd_unix.go:419 +0x40b
net.(*UnixListener).AcceptUnix(0xc2085000c0, 0xc208f11980, 0x0, 0x0)
    /usr/lib/golang/src/net/unixsock_posix.go:282 +0x56
net.(*UnixListener).Accept(0xc2085000c0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/unixsock_posix.go:293 +0x4c
github.com/docker/docker/pkg/listenbuffer.(*defaultListener).Accept(0xc2085000e0, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/listenbuffer/buffer.go:71 +0x67
net/http.(*Server).Serve(0xc20800c840, 0x7f0f2c4c54e0, 0xc2085000e0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:1728 +0x92
github.com/docker/docker/api/server.(*HTTPServer).Serve(0xc2085002a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:113 +0x4d
github.com/docker/docker/api/server.func·006(0xc2085002a0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:86 +0x163
created by github.com/docker/docker/api/server.(*Server).ServeAPI
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:90 +0x144

goroutine 11 [chan receive, 763 minutes]:
github.com/docker/docker/daemon.func·030()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0x5c
created by github.com/docker/docker/daemon.setupDumpStackTrap
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:20 +0x18e

goroutine 15 [select, 5 minutes]:
github.com/docker/libnetwork.(*controller).watchLoop(0xc208741830, 0xc20874abd0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:378 +0x141
created by github.com/docker/libnetwork.(*controller).startWatch
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:392 +0xd3

goroutine 19 [IO wait, 763 minutes]:
net.(*pollDesc).Wait(0xc2084dfa30, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2084dfa30, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readMsg(0xc2084df9d0, 0xc208772170, 0x10, 0x10, 0xc208765220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
    /usr/lib/golang/src/net/fd_unix.go:296 +0x54e
net.(*UnixConn).ReadMsgUnix(0xc208040800, 0xc208772170, 0x10, 0x10, 0xc208765220, 0x1000, 0x1000, 0x51, 0xc20874df34, 0x4, ...)
    /usr/lib/golang/src/net/unixsock_posix.go:147 +0x167
github.com/godbus/dbus.(*oobReader).Read(0xc208765200, 0xc208772170, 0x10, 0x10, 0xc208765200, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
io.ReadAtLeast(0x7f0f2c4c5fa0, 0xc208765200, 0xc208772170, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:298 +0xf1
io.ReadFull(0x7f0f2c4c5fa0, 0xc208765200, 0xc208772170, 0x10, 0x10, 0x51, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:316 +0x6d
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc20874c4a0, 0xc20874ae40, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
github.com/godbus/dbus.(*Conn).inWorker(0xc20873eea0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/conn.go:248 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/auth.go:118 +0xe84

goroutine 20 [chan receive, 763 minutes]:
github.com/godbus/dbus.(*Conn).outWorker(0xc20873eea0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/conn.go:370 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/godbus/dbus/auth.go:119 +0xea1

goroutine 21 [chan receive, 763 minutes]:
github.com/docker/libnetwork/iptables.signalHandler()
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/iptables/firewalld.go:92 +0x57
created by github.com/docker/libnetwork/iptables.FirewalldInit
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/iptables/firewalld.go:48 +0x185

goroutine 33 [IO wait, 763 minutes]:
net.(*pollDesc).Wait(0xc20875dc60, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20875dc60, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20875dc00, 0x0, 0x7f0f2f179500, 0xc208773d68)
    /usr/lib/golang/src/net/fd_unix.go:419 +0x40b
net.(*UnixListener).AcceptUnix(0xc208775e00, 0x10, 0x0, 0x0)
    /usr/lib/golang/src/net/unixsock_posix.go:282 +0x56
net.(*UnixListener).Accept(0xc208775e00, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/unixsock_posix.go:293 +0x4c
github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc208741830, 0xc20875db90, 0x63, 0x7f0f2c4c44e8, 0xc208775e00)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey.go:138 +0x87
created by github.com/docker/libnetwork.(*controller).startExternalKeyListener
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey.go:132 +0x295

goroutine 56 [chan receive, 763 minutes]:
database/sql.(*DB).connectionOpener(0xc2087b8dc0)
    /usr/lib/golang/src/database/sql/sql.go:589 +0x4c
created by database/sql.Open
    /usr/lib/golang/src/database/sql/sql.go:452 +0x31c

goroutine 57 [chan receive]:
github.com/docker/docker/daemon.(*statsCollector).run(0xc2087f5aa0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/stats_collector_unix.go:91 +0xb2
created by github.com/docker/docker/daemon.newStatsCollector
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/stats_collector_unix.go:31 +0x116

goroutine 58 [chan receive, 3 minutes]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc208089540)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/exec.go:300 +0x8c
created by github.com/docker/docker/daemon.NewDaemon
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/daemon.go:833 +0x2be3

goroutine 59 [chan receive, 763 minutes]:
github.com/docker/docker/pkg/signal.func·002()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:29 +0x8f
created by github.com/docker/docker/pkg/signal.Trap
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:55 +0x250

goroutine 110 [semacquire]:
sync.(*Cond).Wait(0xc208aede98)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
github.com/docker/docker/pkg/ioutils.(*bufReader).Read(0xc208aede60, 0xc208afb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:121 +0x18c
bufio.(*Reader).fill(0xc208af55c0)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208af55c0, 0xc20000000a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadBytes(0xc208af55c0, 0xc20c00f70a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:374 +0xd2
github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc20850b300, 0x137b330, 0x6, 0x7f0f2c4daf20, 0xc208aede60)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:47 +0x96
created by github.com/docker/docker/daemon/logger.(*Copier).Run
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:38 +0x11c

goroutine 107 [syscall, 761 minutes]:
syscall.Syscall6(0x3d, 0xc34, 0xc208b6b3ec, 0x0, 0xc208b72630, 0x0, 0x0, 0x4d143c, 0x4d18c2, 0x1121920)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:46 +0x5
syscall.wait4(0xc34, 0xc208b6b3ec, 0x0, 0xc208b72630, 0x90, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:124 +0x79
syscall.Wait4(0xc34, 0xc208b6b434, 0x0, 0xc208b72630, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_linux.go:224 +0x60
os.(*Process).wait(0xc2088316a0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/os/exec_unix.go:22 +0x103
os.(*Process).Wait(0xc2088316a0, 0xc208040af0, 0x0, 0x0)
    /usr/lib/golang/src/os/doc.go:45 +0x3a
os/exec.(*Cmd).Wait(0xc20885ab40, 0x0, 0x0)
    /usr/lib/golang/src/os/exec/exec.go:364 +0x23c
github.com/opencontainers/runc/libcontainer.(*initProcess).wait(0xc208ad9d10, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/opencontainers/runc/libcontainer/process_linux.go:237 +0x3d
github.com/opencontainers/runc/libcontainer.Process.Wait(0xc2087e93d0, 0x1, 0x1, 0xc208867800, 0x8, 0x8, 0x1cd2a58, 0x0, 0x1cd2a58, 0x0, ...)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/opencontainers/runc/libcontainer/process.go:60 +0x11d
github.com/opencontainers/runc/libcontainer.Process.Wait·fm(0xc208b6b9e8, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/execdriver/native/driver.go:179 +0x58
github.com/docker/docker/daemon/execdriver/native.(*Driver).Run(0xc2087df9f0, 0xc2088dfc00, 0xc208aeba10, 0xc2088d1b98, 0x1, 0x1, 0xc2087e8f50, 0x0, 0x0, 0x0, ...)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/execdriver/native/driver.go:185 +0xa2c
github.com/docker/docker/daemon.(*Daemon).run(0xc208089540, 0xc2087d8200, 0xc208aeba10, 0xc2087e8f50, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/daemon.go:956 +0x246
github.com/docker/docker/daemon.(*containerMonitor).Start(0xc20886fc00, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/monitor.go:145 +0x484
github.com/docker/docker/daemon.*containerMonitor.Start·fm(0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/container.go:773 +0x39
github.com/docker/docker/pkg/promise.func·001()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/promise/promise.go:8 +0x2f
created by github.com/docker/docker/pkg/promise.Go
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/promise/promise.go:9 +0xfb

goroutine 32848 [semacquire, 1 minutes]:
sync.(*Cond).Wait(0xc208bd9170)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc208bd9140, 0xc20a4f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209068d68, 0xc20a4f6000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20b6167c8, 0xc20a4f6000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc209cffd80, 0xc20a4f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20951c540)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20951c540, 0xc209c1ed0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20951c540, 0x0, 0x0, 0x0, 0xc20b616800, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc209a1e390, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc209a1e390, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20951c540, 0xc20a7b28f0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20b616780, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20b616780)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 32511 [IO wait, 2 minutes]:
net.(*pollDesc).Wait(0xc20a63ff00, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a63ff00, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a63fea0, 0xc20a9ae000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20a2bc2c8)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2090694c8, 0xc20a9ae000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209069538, 0x7f0f2c4d5460, 0xc2090694c8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 91 [IO wait]:
net.(*pollDesc).Wait(0xc2088e75d0, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2088e75d0, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2088e7570, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x7f0f2f179500, 0xc20ac3bd70)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2088d02a8, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
net/http.(*liveSwitchReader).Read(0xc2088d22c8, 0xc20800f000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc208855780, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc208d9f440)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208d9f440, 0xc20b31060a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc208d9f440, 0x0, 0x0, 0x0, 0xc20bed2400, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc20bf6dce0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc20bf6dce0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc208d9f440, 0xc208b7c750, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc2088d2280, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc2088d2280)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 109 [semacquire, 761 minutes]:
sync.(*Cond).Wait(0xc208508ff0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc208508fc0, 0xc2088ca400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2088d1b80, 0xc2088ca400, 0x400, 0x400, 0xf3a280, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
github.com/docker/docker/pkg/ioutils.(*bufReader).drain(0xc208aedef0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:90 +0x74
created by github.com/docker/docker/pkg/ioutils.NewBufReader
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:69 +0x351

goroutine 108 [semacquire]:
sync.(*Cond).Wait(0xc208508f30)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc208508f00, 0xc2088ca000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2088d1b70, 0xc2088ca000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
github.com/docker/docker/pkg/ioutils.(*bufReader).drain(0xc208aede60)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:90 +0x74
created by github.com/docker/docker/pkg/ioutils.NewBufReader
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:69 +0x351

goroutine 117 [chan receive, 761 minutes]:
github.com/docker/docker/daemon.wait(0xc208b21e00, 0xffffffffc4653600, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/state.go:110 +0x71
github.com/docker/docker/daemon.(*State).WaitStop(0xc2088cd340, 0xffffffffc4653600, 0x40, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/state.go:152 +0xc7
github.com/docker/docker/daemon.(*Daemon).ContainerWait(0xc208089540, 0xc2088497b7, 0x40, 0xffffffffc4653600, 0xc208b09dd8, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/wait.go:16 +0x9c
github.com/docker/docker/api/server/router/local.(*router).postContainersWait(0xc2087ffb00, 0x7f0f2c4d5590, 0xc208b82960, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/container.go:271 +0xbb
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.postContainersWait)·fm(0x7f0f2c4d5590, 0xc208b82960, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:136 +0x7b
github.com/docker/docker/api/server.func·004(0x7f0f2c4d5590, 0xc208b82960, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520, 0xc208b82840, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:161 +0x28a
net/http.HandlerFunc.ServeHTTP(0xc2080aeec0, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520)
    /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2087dfef0, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20800c840, 0x7f0f2c4d5558, 0xc208b98140, 0xc208b7d520)
    /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208b980a0)
    /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 111 [semacquire, 761 minutes]:
sync.(*Cond).Wait(0xc208aedf28)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
github.com/docker/docker/pkg/ioutils.(*bufReader).Read(0xc208aedef0, 0xc208afe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/ioutils/readers.go:121 +0x18c
bufio.(*Reader).fill(0xc208af5620)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208af5620, 0x4abc0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadBytes(0xc208af5620, 0xc208aede0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:374 +0xd2
github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc20850b300, 0x137b2f0, 0x6, 0x7f0f2c4daf20, 0xc208aedef0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:47 +0x96
created by github.com/docker/docker/daemon/logger.(*Copier).Run
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/logger/copier.go:38 +0x11c

goroutine 112 [syscall]:
syscall.Syscall(0x0, 0x19, 0xc208b56000, 0x8000, 0x0, 0x8000, 0x0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.read(0x19, 0xc208b56000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:867 +0x6e
syscall.Read(0x19, 0xc208b56000, 0x8000, 0x8000, 0xd6, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_unix.go:136 +0x58
os.(*File).read(0xc2088d1c18, 0xc208b56000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/os/file_unix.go:191 +0x5e
os.(*File).Read(0xc2088d1c18, 0xc208b56000, 0x8000, 0x8000, 0xd6, 0x0, 0x0)
    /usr/lib/golang/src/os/file.go:95 +0x91
io.Copy(0x7f0f2c4daf48, 0xc208793940, 0x7f0f2f1794b0, 0xc2088d1c18, 0x1cba4, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
os/exec.func·003(0x0, 0x0)
    /usr/lib/golang/src/os/exec/exec.go:221 +0x7d
os/exec.func·004(0xc2088314c0)
    /usr/lib/golang/src/os/exec/exec.go:328 +0x2d
created by os/exec.(*Cmd).Start
    /usr/lib/golang/src/os/exec/exec.go:329 +0xb6d

goroutine 113 [syscall, 761 minutes]:
syscall.Syscall(0x0, 0x1b, 0xc208b5e000, 0x8000, 0x0, 0x8000, 0x0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.read(0x1b, 0xc208b5e000, 0x8000, 0x8000, 0x1cc8120, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:867 +0x6e
syscall.Read(0x1b, 0xc208b5e000, 0x8000, 0x8000, 0xf64a00, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_unix.go:136 +0x58
os.(*File).read(0xc2088d1c38, 0xc208b5e000, 0x8000, 0x8000, 0xc208b5e000, 0x0, 0x0)
    /usr/lib/golang/src/os/file_unix.go:191 +0x5e
os.(*File).Read(0xc2088d1c38, 0xc208b5e000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/lib/golang/src/os/file.go:95 +0x91
io.Copy(0x7f0f2c4daf48, 0xc208793920, 0x7f0f2f1794b0, 0xc2088d1c38, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
os/exec.func·003(0x0, 0x0)
    /usr/lib/golang/src/os/exec/exec.go:221 +0x7d
os/exec.func·004(0xc208831500)
    /usr/lib/golang/src/os/exec/exec.go:328 +0x2d
created by os/exec.(*Cmd).Start
    /usr/lib/golang/src/os/exec/exec.go:329 +0xb6d

goroutine 114 [select]:
github.com/docker/libnetwork/osl.removeUnusedPaths()
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:73 +0x48b
created by github.com/docker/libnetwork/osl.createBasePath
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:58 +0xb1

goroutine 115 [syscall, 761 minutes]:
syscall.Syscall(0x0, 0x17, 0xc20881df50, 0x8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.read(0x17, 0xc20881df50, 0x8, 0x8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:867 +0x6e
syscall.Read(0x17, 0xc20881df50, 0x8, 0x8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_unix.go:136 +0x58
os.(*File).read(0xc208040aa8, 0xc20881df50, 0x8, 0x8, 0xc20810ce00, 0x0, 0x0)
    /usr/lib/golang/src/os/file_unix.go:191 +0x5e
os.(*File).Read(0xc208040aa8, 0xc20881df50, 0x8, 0x8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/os/file.go:95 +0x91
github.com/opencontainers/runc/libcontainer.func·009()
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/opencontainers/runc/libcontainer/notify_linux.go:51 +0x18c
created by github.com/opencontainers/runc/libcontainer.notifyOnOOM
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/opencontainers/runc/libcontainer/notify_linux.go:61 +0x887

goroutine 116 [chan receive, 761 minutes]:
github.com/docker/docker/daemon.func·036()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/monitor.go:250 +0x49
created by github.com/docker/docker/daemon.(*containerMonitor).callback
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/monitor.go:254 +0x100

goroutine 127 [select]:
github.com/docker/docker/api/server/router/local.(*router).getEvents(0xc2087ffb00, 0x7f0f2c4d5590, 0xc208b83c80, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/info.go:123 +0x841
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.getEvents)·fm(0x7f0f2c4d5590, 0xc208b83c80, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:100 +0x7b
github.com/docker/docker/api/server.func·004(0x7f0f2c4d5590, 0xc208b83c80, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680, 0xc208b83bc0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:161 +0x28a
net/http.HandlerFunc.ServeHTTP(0xc208830740, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680)
    /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2087dfef0, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20800c840, 0x7f0f2c4d5558, 0xc208b99360, 0xc208b9e680)
    /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208b992c0)
    /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 128 [IO wait, 761 minutes]:
net.(*pollDesc).Wait(0xc20883cfb0, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20883cfb0, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20883cf50, 0xc208ba6000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20874d408)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208040e20, 0xc208ba6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc208040e90, 0x7f0f2c4d5460, 0xc208040e20, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 132 [select]:
github.com/docker/docker/api/server/router/local.(*router).getEvents(0xc2087ffb00, 0x7f0f2c4d5590, 0xc208bb0480, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/info.go:123 +0x841
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.getEvents)·fm(0x7f0f2c4d5590, 0xc208bb0480, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:100 +0x7b
github.com/docker/docker/api/server.func·004(0x7f0f2c4d5590, 0xc208bb0480, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30, 0xc208bb03c0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:161 +0x28a
net/http.HandlerFunc.ServeHTTP(0xc208830740, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30)
    /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2087dfef0, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20800c840, 0x7f0f2c4d5558, 0xc208b99ae0, 0xc208b9ec30)
    /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208b99a40)
    /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 133 [IO wait, 761 minutes]:
net.(*pollDesc).Wait(0xc20883d870, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20883d870, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20883d810, 0xc208bbc000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20874dfe0)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208040f60, 0xc208bbc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc208040fd0, 0x7f0f2c4d5460, 0xc208040f60, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 33245 [semacquire, 1 minutes]:
sync.(*Cond).Wait(0xc209af03f0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc209af03c0, 0xc208a59000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209068410, 0xc208a59000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20add5da8, 0xc208a59000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc20b6dd4a0, 0xc208a59000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20a267aa0)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20a267aa0, 0xc20ada6a0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20a267aa0, 0x0, 0x0, 0x0, 0xc20add5e00, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc209972ea0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc209972ea0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20a267aa0, 0xc2098d49c0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20add5d60, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20add5d60)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34050 [semacquire]:
sync.(*Cond).Wait(0xc2096c4930)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc2096c4900, 0xc2087ca000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2090687a8, 0xc2087ca000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
io/ioutil.(*nopCloser).Read(0xc208b936d0, 0xc2087ca000, 0x8000, 0x8000, 0x7f0f2c4d59a0, 0x0, 0x0)
    <autogenerated>:4 +0x89
github.com/docker/docker/pkg/progressreader.(*Config).Read(0xc20b6d9810, 0xc2087ca000, 0x8000, 0x8000, 0x35f, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/progressreader/progressreader.go:31 +0x8e
io.Copy(0x7f0f2c4d59f0, 0xc208b754a0, 0x7f0f2c4e2980, 0xc20b6d9810, 0x177f600, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
github.com/docker/docker/graph.func·016()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:270 +0xb9
created by github.com/docker/docker/graph.(*v2Pusher).pushV2Image
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:282 +0xa54

goroutine 32510 [semacquire, 2 minutes]:
sync.(*Cond).Wait(0xc209af10b0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc209af1080, 0xc208c6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209069530, 0xc208c6c000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20914f808, 0xc208c6c000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc2091d27a0, 0xc208c6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20894f5c0)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20894f5c0, 0xc209868e0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20894f5c0, 0x0, 0x0, 0x0, 0xc20914f900, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc20ab697a0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc20ab697a0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20894f5c0, 0xc208ba08f0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20914f7c0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20914f7c0)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 33470 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc20b6d6d80, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b6d6d80, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b6d6d20, 0xc209d26000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20988fa20)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2090685f0, 0xc209d26000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc2090686a8, 0x7f0f2c4d5460, 0xc2090685f0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 34038 [semacquire]:
sync.(*Cond).Wait(0xc2096c4b70)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc2096c4b40, 0xc2090ca000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209068850, 0xc2090ca000, 0x8000, 0x8000, 0x35f, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
io.Copy(0x7f0f2c4d5a70, 0xc2090687b0, 0x7f0f2c4d5ae8, 0xc209068850, 0x177f600, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
github.com/docker/docker/graph.func·007()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/graph.go:828 +0x867
created by github.com/docker/docker/graph.(*Graph).assembleTarLayer
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/graph.go:833 +0x512

goroutine 33469 [semacquire, 1 minutes]:
sync.(*Cond).Wait(0xc209af0b70)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc209af0b40, 0xc20895f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2090686a0, 0xc20895f000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20a239948, 0xc20895f000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc209b4dfe0, 0xc20895f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc209115740)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc209115740, 0xc209510e0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc209115740, 0x0, 0x0, 0x0, 0xc20a239900, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc208f0c720, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc208f0c720, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc209115740, 0xc208ba1a00, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20a239900, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20a239900)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 30815 [select]:
net/http.(*persistConn).roundTrip(0xc208b75600, 0xc208b93730, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc209f12c60, 0xc20a7b2680, 0xc20a7b2680, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:235 +0x558
github.com/docker/distribution/registry/client/transport.(*transport).RoundTrip(0xc208cd8c40, 0xc20a7b24e0, 0xc2089f38f0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/distribution/registry/client/transport/transport.go:62 +0x20b
net/http.send(0xc20a7b24e0, 0x7f0f2c4dc5b0, 0xc208cd8c40, 0x4aa179, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0xc20c0f6150, 0xc20a7b24e0, 0xc208834700, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:142 +0x15b
net/http.(*Client).Do(0xc20c0f6150, 0xc20a7b24e0, 0xc20a9812c0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:179 +0x220
github.com/docker/distribution/registry/client.(*httpBlobUpload).ReadFrom(0xc20a266ae0, 0x7f0f2c4d5ae8, 0xc2090689c8, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/docker/distribution/registry/client/blob_writer.go:42 +0x24e
github.com/docker/docker/graph.(*v2Pusher).pushV2Image(0xc209065ea0, 0x7f0f244ce428, 0xc209390e00, 0xc20bf02000, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:285 +0xba5
github.com/docker/docker/graph.(*v2Pusher).pushV2Tag(0xc209065ea0, 0xc20b05e499, 0xc, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:177 +0x130e
github.com/docker/docker/graph.(*v2Pusher).pushV2Repository(0xc209065ea0, 0xc20b05e499, 0xc, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:83 +0x5cf
github.com/docker/docker/graph.(*v2Pusher).Push(0xc209065ea0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push_v2.go:47 +0x288
github.com/docker/docker/graph.(*TagStore).Push(0xc20800d1a0, 0xc20b05e473, 0x1c, 0xc20bf91b30, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/push.go:108 +0x9a5
github.com/docker/docker/daemon.(*Daemon).PushImage(0xc208089540, 0xc20b05e473, 0x1c, 0xc20bf91b30, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/daemon.go:1067 +0x5c
github.com/docker/docker/api/server/router/local.(*router).postImagesPush(0xc2087ffb00, 0x7f0f2c4d5590, 0xc20bf91a10, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/image.go:196 +0xa54
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.postImagesPush)·fm(0x7f0f2c4d5590, 0xc20bf91a10, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:127 +0x7b
github.com/docker/docker/api/server.func·004(0x7f0f2c4d5590, 0xc20bf91a10, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0, 0xc20bf91830, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:161 +0x28a
net/http.HandlerFunc.ServeHTTP(0xc2088bdb00, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0)
    /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2087dfef0, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20800c840, 0x7f0f2c4d5558, 0xc209065cc0, 0xc20a76d1e0)
    /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc209065c20)
    /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 32682 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc209e91b80, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc209e91b80, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc209e91b20, 0xc20ae4a000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc2087dbb60)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209068ed8, 0xc20ae4a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209068f98, 0x7f0f2c4d5460, 0xc209068ed8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 34661 [IO wait]:
net.(*pollDesc).Wait(0xc208a50140, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208a50140, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208a500e0, 0xc209afe000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc208ad0080)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208040d88, 0xc209afe000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc208040e48, 0x7f0f2c4d5460, 0xc208040d88, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 32541 [IO wait, 2 minutes]:
net.(*pollDesc).Wait(0xc20b6d7870, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b6d7870, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b6d7810, 0xc209180000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc209c1a818)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209068ef8, 0xc209180000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209068f88, 0x7f0f2c4d5460, 0xc209068ef8, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 34053 [IO wait]:
net.(*pollDesc).Wait(0xc20b6d9aa0, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b6d9aa0, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b6d9a40, 0xc20aec0000, 0x1000, 0x1000, 0x0, 0x7f0f2f179500, 0xc208b93c40)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209068a70, 0xc20aec0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x7f0f2c4dc750, 0xc209068a70, 0xc208b75658, 0xc20aec0000, 0x1000, 0x1000, 0x7f0f2f167010, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20a81c320, 0xc20aec0000, 0x1000, 0x1000, 0xc207ef9757, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20a267b60)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20a267b60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208b75600)
    /usr/lib/golang/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:660 +0xc9f

goroutine 30793 [runnable]:
net.(*pollDesc).Wait(0xc20875ca70, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20875ca70, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20875ca10, 0xc2089cb000, 0x1000, 0x1000, 0x0, 0x7f0f2f179500, 0xc208b92468)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2088d1818, 0xc2089cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
net/http.(*liveSwitchReader).Read(0xc20af5eb88, 0xc2089cb000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc20b5ad500, 0xc2089cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20b2892c0)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20b2892c0, 0xc20a8ce10a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20b2892c0, 0x0, 0x0, 0x0, 0xc207fb0800, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc2084f7b60, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc2084f7b60, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20b2892c0, 0xc208b7cc30, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20af5eb40, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20af5eb40)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 32681 [semacquire, 1 minutes]:
sync.(*Cond).Wait(0xc20ab90ab0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc20ab90a80, 0xc209b25000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209068f90, 0xc209b25000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc2088f4368, 0xc209b25000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc2097373e0, 0xc209b25000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc208c4dec0)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208c4dec0, 0xc20affbc0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc208c4dec0, 0x0, 0x0, 0x0, 0xc2088f4300, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc208d39ec0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc208d39ec0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc208c4dec0, 0xc20a76cd00, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc2088f4320, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc2088f4320)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34653 [syscall]:
syscall.Syscall(0x10, 0x11, 0x1277, 0xc20a80ca00, 0xc208834380, 0xb46a67, 0xc20a80ca18)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:21 +0x5
github.com/docker/docker/pkg/devicemapper.ioctlBlkDiscard(0x11, 0x0, 0x1900000000, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/devicemapper/ioctl.go:68 +0x7c
github.com/docker/docker/pkg/devicemapper.BlockDeviceDiscard(0xc20b32c070, 0x62, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/pkg/devicemapper/devmapper.go:479 +0x121
github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).issueDiscard(0xc2080fa300, 0xc20ac374a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:1754 +0x4b6
github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).deleteDevice(0xc2080fa300, 0xc20ac374a0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:1763 +0x59
github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).DeleteDevice(0xc2080fa300, 0xc209391580, 0x40, 0xc20b146f00, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:1805 +0x5aa
github.com/docker/docker/daemon/graphdriver/devmapper.(*Driver).Remove(0xc20873a5a0, 0xc209391580, 0x40, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/driver.go:154 +0xc5
github.com/docker/docker/daemon/graphdriver.(*NaiveDiffDriver).Remove(0xc20872ab40, 0xc209391580, 0x40, 0x0, 0x0)
    <autogenerated>:29 +0x7c
github.com/docker/docker/graph.(*Graph).Delete(0xc2087410e0, 0xc209390fc0, 0x40, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/graph.go:428 +0x1f3
github.com/docker/docker/daemon.(*Daemon).imageDeleteHelper(0xc208089540, 0xc209375a40, 0xc20a807ae0, 0xc209010100, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/image_delete.go:226 +0x1c9
github.com/docker/docker/daemon.(*Daemon).imageDeleteHelper(0xc208089540, 0xc2093756c0, 0xc20a807ae0, 0xc209010100, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/image_delete.go:247 +0x59f
github.com/docker/docker/daemon.(*Daemon).imageDeleteHelper(0xc208089540, 0xc209374fc0, 0xc20a807ae0, 0xc209010100, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/image_delete.go:247 +0x59f
github.com/docker/docker/daemon.(*Daemon).imageDeleteHelper(0xc208089540, 0xc2090c1a40, 0xc20a807ae0, 0x10100, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/image_delete.go:247 +0x59f
github.com/docker/docker/daemon.(*Daemon).ImageDelete(0xc208089540, 0xc208b1e1a5, 0x2d, 0x100, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/image_delete.go:107 +0x7ca
github.com/docker/docker/api/server/router/local.(*router).deleteImages(0xc2087ffb00, 0x7f0f2c4d5590, 0xc20a64db00, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/image.go:249 +0x1d6
github.com/docker/docker/api/server/router/local.*router.(github.com/docker/docker/api/server/router/local.deleteImages)·fm(0x7f0f2c4d5590, 0xc20a64db00, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/router/local/local.go:149 +0x7b
github.com/docker/docker/api/server.func·004(0x7f0f2c4d5590, 0xc20a64db00, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:87 +0x7c7
github.com/docker/docker/api/server.func·003(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:66 +0x10e
github.com/docker/docker/api/server.func·002(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:49 +0x47c
github.com/docker/docker/api/server.func·001(0x7f0f2f179a50, 0xc208032ee8, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820, 0xc20a64d7a0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/middleware.go:27 +0x1dc
github.com/docker/docker/api/server.func·007(0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/api/server/server.go:161 +0x28a
net/http.HandlerFunc.ServeHTTP(0xc20875e080, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820)
    /usr/lib/golang/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc2087dfef0, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820)
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc20800c840, 0x7f0f2c4d5558, 0xc208c4efa0, 0xc208b7c820)
    /usr/lib/golang/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc208c4ef00)
    /usr/lib/golang/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34044 [syscall]:
syscall.Syscall(0x0, 0xe, 0xc20941e000, 0x8000, 0x200000003, 0xc208020a80, 0xfe37c0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.read(0xe, 0xc20941e000, 0x8000, 0x8000, 0x7f0f245ae210, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:867 +0x6e
syscall.Read(0xe, 0xc20941e000, 0x8000, 0x8000, 0xe, 0x0, 0x0)
    /usr/lib/golang/src/syscall/syscall_unix.go:136 +0x58
os.(*File).read(0xc208041360, 0xc20941e000, 0x8000, 0x8000, 0x7f0f2c4de738, 0x0, 0x0)
    /usr/lib/golang/src/os/file_unix.go:191 +0x5e
os.(*File).Read(0xc208041360, 0xc20941e000, 0x8000, 0x8000, 0x85, 0x0, 0x0)
    /usr/lib/golang/src/os/file.go:95 +0x91
io.(*teeReader).Read(0xc20ac241e0, 0xc20941e000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:495 +0x83
io.Copy(0x7f0f2c4d5a70, 0xc209068858, 0x7f0f2c4d5a98, 0xc20ac241e0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
github.com/vbatts/tar-split/tar/asm.func·001()
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/vbatts/tar-split/tar/asm/assemble.go:49 +0x474
created by github.com/vbatts/tar-split/tar/asm.NewOutputTarStream
    /home/ec2-user/build/src/github.com/docker/docker/vendor/src/github.com/vbatts/tar-split/tar/asm/assemble.go:65 +0x17b

goroutine 33246 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc20a262920, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20a262920, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20a2628c0, 0xc209d2e000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc208ad0be0)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209068338, 0xc209d2e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209068418, 0x7f0f2c4d5460, 0xc209068338, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 34054 [semacquire]:
sync.(*Cond).Wait(0xc2096c4f30)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc2096c4f00, 0xc20a518000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2090689c8, 0xc20a518000, 0x8000, 0x8000, 0x13633b0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
io/ioutil.(*nopCloser).Read(0xc208b93710, 0xc20a518000, 0x8000, 0x8000, 0xc20b84eb00, 0x0, 0x0)
    <autogenerated>:4 +0x89
io.(*multiReader).Read(0xc20ac241a0, 0xc20a518000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/lib/golang/src/io/multi.go:13 +0xab
io.Copy(0x7f0f244ce748, 0xc208f0e5f0, 0x7f0f2c4c5fc8, 0xc20ac241a0, 0xbe0000, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.(*transferWriter).WriteBody(0xc20b6d9ab0, 0x7f0f2c4dc7c8, 0xc20b84eb00, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transfer.go:210 +0x226
net/http.(*Request).write(0xc20a7b2680, 0x7f0f2c4dc7c8, 0xc20b84eb00, 0xc208b75400, 0xc2089f3da0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:448 +0xada
net/http.(*persistConn).writeLoop(0xc208b75600)
    /usr/lib/golang/src/net/http/transport.go:951 +0x28d
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:661 +0xcbc

goroutine 34645 [semacquire]:
sync.(*Cond).Wait(0xc2096c5bf0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc2096c5bc0, 0xc20c0da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209069cf0, 0xc20c0da000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc208c4e548, 0xc20c0da000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc20abda8e0, 0xc20c0da000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20b6aed20)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20b6aed20, 0xc208d3c10a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20b6aed20, 0x0, 0x0, 0x0, 0xc208c4e500, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc20c140e10, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc20c140e10, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20b6aed20, 0xc20a6bb2b0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc208c4e500, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc208c4e500)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34660 [semacquire]:
sync.(*Cond).Wait(0xc20ab906f0)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc20ab906c0, 0xc20a7ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc208040e40, 0xc20a7ab000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20bed2548, 0xc20a7ab000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc208f11980, 0xc20a7ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc208cae540)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208cae540, 0xc20b31130a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc208cae540, 0x0, 0x0, 0x0, 0xc20bed2500, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc20ae6e240, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc20ae6e240, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc208cae540, 0xc208b7cb60, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20bed2500, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20bed2500)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34658 [semacquire]:
sync.(*Cond).Wait(0xc20ab90630)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc20ab90600, 0xc20b8f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc2080410d0, 0xc20b8f8000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20a2393a8, 0xc20b8f8000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc20ba71c60, 0xc20b8f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc20a223920)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc20a223920, 0xc20b34ef0a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc20a223920, 0x0, 0x0, 0x0, 0xc20a239400, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc2087803c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc2087803c0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc20a223920, 0xc208b7c680, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20a239360, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20a239360)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 34646 [IO wait]:
net.(*pollDesc).Wait(0xc208750060, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208750060, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208750000, 0xc2089a8000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20abd6a70)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209069c00, 0xc2089a8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209069cf8, 0x7f0f2c4d5460, 0xc209069c00, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 34100 [semacquire]:
sync.(*Mutex).Lock(0xc2080fa308)
    /usr/lib/golang/src/sync/mutex.go:66 +0xd3
github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).lookupDeviceWithLock(0xc2080fa300, 0xc20aaede00, 0x40, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:396 +0x53
github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).MountDevice(0xc2080fa300, 0xc20aaede00, 0x40, 0xc20a63e540, 0x61, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:1994 +0x68
github.com/docker/docker/daemon/graphdriver/devmapper.(*Driver).Get(0xc20873a5a0, 0xc20aaede00, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/driver.go:183 +0x440
github.com/docker/docker/daemon/graphdriver.(*NaiveDiffDriver).Get(0xc20872ab40, 0xc20aaede00, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    <autogenerated>:26 +0xb1
github.com/docker/docker/graph.func·007()
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/graph.go:816 +0x407
created by github.com/docker/docker/graph.(*Graph).assembleTarLayer
    /home/ec2-user/build/src/github.com/docker/docker/.gopath/src/github.com/docker/docker/graph/graph.go:833 +0x512

goroutine 34659 [IO wait]:
net.(*pollDesc).Wait(0xc208a50840, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208a50840, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208a507e0, 0xc208cf6000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc20b9a0350)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208041010, 0xc208cf6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc2080410d8, 0x7f0f2c4d5460, 0xc208041010, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

goroutine 32540 [semacquire, 2 minutes]:
sync.(*Cond).Wait(0xc20ab90930)
    /usr/lib/golang/src/sync/cond.go:62 +0x9e
io.(*pipe).read(0xc20ab90900, 0xc208904000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:52 +0x303
io.(*PipeReader).Read(0xc209068f80, 0xc208904000, 0x1000, 0x1000, 0x7f0f2c4c4400, 0x0, 0x0)
    /usr/lib/golang/src/io/pipe.go:134 +0x5b
net/http.(*liveSwitchReader).Read(0xc20ab8cfe8, 0xc208904000, 0x1000, 0x1000, 0x57865e, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:214 +0xab
io.(*LimitedReader).Read(0xc208749720, 0xc208904000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:408 +0xce
bufio.(*Reader).fill(0xc208c4cf60)
    /usr/lib/golang/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).ReadSlice(0xc208c4cf60, 0xc2080b340a, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:295 +0x257
bufio.(*Reader).ReadLine(0xc208c4cf60, 0x0, 0x0, 0x0, 0xc20ab8d200, 0x0, 0x0)
    /usr/lib/golang/src/bufio/bufio.go:324 +0x62
net/textproto.(*Reader).readLineSlice(0xc209ff0810, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:55 +0x9e
net/textproto.(*Reader).ReadLine(0xc209ff0810, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/textproto/reader.go:36 +0x4f
net/http.ReadRequest(0xc208c4cf60, 0xc20a76dad0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/request.go:598 +0xcb
net/http.(*conn).readRequest(0xc20ab8cfa0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/server.go:586 +0x26f
net/http.(*conn).serve(0xc20ab8cfa0)
    /usr/lib/golang/src/net/http/server.go:1162 +0x69e
created by net/http.(*Server).Serve
    /usr/lib/golang/src/net/http/server.go:1751 +0x35e

goroutine 32849 [IO wait, 1 minutes]:
net.(*pollDesc).Wait(0xc20b382840, 0x72, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20b382840, 0x0, 0x0)
    /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc20b3827e0, 0xc208a40000, 0x8000, 0x8000, 0x0, 0x7f0f2f179500, 0xc2097c6e88)
    /usr/lib/golang/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc209068cc0, 0xc208a40000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:121 +0xdc
io.Copy(0x7f0f2c4d5a70, 0xc209068d80, 0x7f0f2c4d5460, 0xc209068cc0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/io/io.go:362 +0x1f6
net/http.func·008()
    /usr/lib/golang/src/net/http/server.go:171 +0x7d
created by net/http.(*conn).closeNotify
    /usr/lib/golang/src/net/http/server.go:177 +0x2e2

The text was updated successfully, but these errors were encountered:

tiborvass · 2015-11-02T20:20:15Z

@alexmac big thanks for the report

stevvooe · 2015-11-02T20:22:48Z

Looks like the classic issue of handing an http request body without properly wrapping the closer.

tonistiigi · 2015-11-02T21:06:12Z

@stevvooe So what solution are you proposing? Wrapping out through io.Pipe like the pull side used to do?

stevvooe · 2015-11-02T21:35:47Z

@tonistiigi I am not sure. The "poisoned" reader needs to be tracked down and wrapped with closer that intercepts calls to closed resources.

dmcgowan · 2015-11-02T23:52:00Z

This panic appears to be related to 4dce280. There is a race betwee pipeReader.Close() and the closing of the Outstream by the caller which is used by the progress reader during io.Copy. Since previous to this change the io.Copy would have guaranteed to be ended before an error was returned this would not panic. Now there is nothing guaranteeing that the io.Copy has completed before returning error to the caller. Since this goroutine (4dce280#diff-592041b611364e78f016f9bce43e90a7R262) is not intended to outlast the calling function I think it best to add some synchronization around that goroutine.

tiborvass added this to the 1.9.0 milestone Nov 2, 2015

stevvooe mentioned this issue Nov 3, 2015

Avoid panic on write after close in http #17634

Merged

tiborvass modified the milestones: 1.9.1, 1.9.0 Nov 3, 2015

LK4D4 closed this as completed in #17634 Nov 3, 2015

stevvooe mentioned this issue Nov 7, 2015

Docker runtime error #17759

Closed

tonistiigi mentioned this issue Nov 16, 2015

Docker panic on push to registry v2 #18012

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker daemon panic #17625

docker daemon panic #17625

alexmac commented Nov 2, 2015

tiborvass commented Nov 2, 2015

stevvooe commented Nov 2, 2015

tonistiigi commented Nov 2, 2015

stevvooe commented Nov 2, 2015

dmcgowan commented Nov 2, 2015

docker daemon panic #17625

docker daemon panic #17625

Comments

alexmac commented Nov 2, 2015

tiborvass commented Nov 2, 2015

stevvooe commented Nov 2, 2015

tonistiigi commented Nov 2, 2015

stevvooe commented Nov 2, 2015

dmcgowan commented Nov 2, 2015