net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2)

[clnperez]

time="2016-01-18T16:31:37.532208479Z" level=debug msg="POST /v1.22/build?buildargs=%7B%7D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&shmsize=0&t=testbuildaddnotfound&ulimits=null" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x7a9ad1]

goroutine 2807 [running]:
net/http.startCloseNotifyBackgroundRead(0xc820e1c080, 0xc8217cc180)
        /usr/local/go/src/net/http/server.go:1560 +0x21
net/http.(*response).CloseNotify(0xc820fe8600, 0x0)
        /usr/local/go/src/net/http/server.go:1553 +0x2a5
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild.func4(0xc82150e3c0, 0x7f50a0304f58, 0xc820fe8600, 0xc820b195f0)
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:247 +0x39
created by github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:251 +0x14ca

I've traced this error back to commit 99fb191

The docker image itself is running Ubuntu trusty:
docker run -it --privileged docker-go1.6beta2 uname -a
Linux 4b2dfe4ce10b 4.2.8-200.fc22.x86_64 #1 SMP Tue Dec 15 16:50:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here's how to get the docker env set up to recreate (all in docker images).
0) Install docker

1. git clone https://github.com/docker/docker.git && cd docker
2. edit Dockerfile:
   -ENV GO_VERSION 1.5.3
   +ENV GO_VERSION 1.6beta2
3. docker build -t docker .
4. docker run -it --privileged -v pwd:/go/src/github.com/docker/docker/ ./hack/make.sh binary test-integration-cli

You'll see all the docker tests start to fail with "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

The -v flag will volume-mount your local directory into the container, so then you can look at logs locally after the container exits. Check less bundles/latest/test-integration-cli/docker.log for the stack-trace I pasted above.

Because of how the tests are run it's not trivial to tell which of the tests causes the panic that kills the docker daemon, but I'm working on that.

[ianlancetaylor]

CC @bradfitz

[laboger]

fails on x86 and ppc64le

[bradfitz]

Docker wasn't using CloseNotify from the right goroutine. Patch for Docker:

diff --git a/api/server/server.go b/api/server/server.go
index a1c57a8..e6a2957 100644
--- a/api/server/server.go
+++ b/api/server/server.go
@@ -1091,10 +1091,11 @@ func postBuild(eng *engine.Engine, version version.Version, w http.ResponseWrite
        if closeNotifier, ok := w.(http.CloseNotifier); ok {
                finished := make(chan struct{})
                defer close(finished)
+               clientGone := closeNotifier.CloseNotify()
                go func() {
                        select {
                        case <-finished:
-                       case <-closeNotifier.CloseNotify():
+                       case <-clientGone:
                                log.Infof("Client disconnected, cancelling job: %v", job)
                                job.Cancel()
                        }

If it worked before, it was just luck. Or it was racy. We've improved the CloseNotifier docs a bunch in this cycle.

But I'll make Go panic with a more useful error message so you and others can diagnose this earlier and more easily in the future.

[gopherbot]

CL https://golang.org/cl/18708 mentions this issue.

[clnperez]

Thanks. It looks like that's the only spot in the docker code that's an issue, so I can submit a PR to them for this.

[cpuguy83]

<3 Thanks. Probably haven't seen it before b/c we haven't done a release with GOMAXPROCS set higher than 1.