Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) #14001

Closed
clnperez opened this issue Jan 18, 2016 · 6 comments

Comments

Projects
None yet
6 participants
@clnperez
Copy link

commented Jan 18, 2016

time="2016-01-18T16:31:37.532208479Z" level=debug msg="POST /v1.22/build?buildargs=%7B%7D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&shmsize=0&t=testbuildaddnotfound&ulimits=null" 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x7a9ad1]

goroutine 2807 [running]:
net/http.startCloseNotifyBackgroundRead(0xc820e1c080, 0xc8217cc180)
        /usr/local/go/src/net/http/server.go:1560 +0x21
net/http.(*response).CloseNotify(0xc820fe8600, 0x0)
        /usr/local/go/src/net/http/server.go:1553 +0x2a5
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild.func4(0xc82150e3c0, 0x7f50a0304f58, 0xc820fe8600, 0xc820b195f0)
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:247 +0x39
created by github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild
        /go/src/github.com/docker/docker/api/server/router/build/build_routes.go:251 +0x14ca

I've traced this error back to commit 99fb191

The docker image itself is running Ubuntu trusty:
docker run -it --privileged docker-go1.6beta2 uname -a
Linux 4b2dfe4ce10b 4.2.8-200.fc22.x86_64 #1 SMP Tue Dec 15 16:50:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Here's how to get the docker env set up to recreate (all in docker images).
0) Install docker

  1. git clone https://github.com/docker/docker.git && cd docker
  2. edit Dockerfile:
    -ENV GO_VERSION 1.5.3
    +ENV GO_VERSION 1.6beta2
  3. docker build -t docker .
  4. docker run -it --privileged -v pwd:/go/src/github.com/docker/docker/ ./hack/make.sh binary test-integration-cli

You'll see all the docker tests start to fail with "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

The -v flag will volume-mount your local directory into the container, so then you can look at logs locally after the container exits. Check less bundles/latest/test-integration-cli/docker.log for the stack-trace I pasted above.

Because of how the tests are run it's not trivial to tell which of the tests causes the panic that kills the docker daemon, but I'm working on that.

@ianlancetaylor ianlancetaylor changed the title "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) net/http: "panic: runtime error: invalid memory address or nil pointer dereference" when running docker tests (docker built with 1.6beta2) Jan 18, 2016

@ianlancetaylor ianlancetaylor added this to the Go1.6 milestone Jan 18, 2016

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2016

@laboger

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2016

fails on x86 and ppc64le

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 18, 2016

Docker wasn't using CloseNotify from the right goroutine. Patch for Docker:

diff --git a/api/server/server.go b/api/server/server.go
index a1c57a8..e6a2957 100644
--- a/api/server/server.go
+++ b/api/server/server.go
@@ -1091,10 +1091,11 @@ func postBuild(eng *engine.Engine, version version.Version, w http.ResponseWrite
        if closeNotifier, ok := w.(http.CloseNotifier); ok {
                finished := make(chan struct{})
                defer close(finished)
+               clientGone := closeNotifier.CloseNotify()
                go func() {
                        select {
                        case <-finished:
-                       case <-closeNotifier.CloseNotify():
+                       case <-clientGone:
                                log.Infof("Client disconnected, cancelling job: %v", job)
                                job.Cancel()
                        }

If it worked before, it was just luck. Or it was racy. We've improved the CloseNotifier docs a bunch in this cycle.

But I'll make Go panic with a more useful error message so you and others can diagnose this earlier and more easily in the future.

@gopherbot

This comment has been minimized.

Copy link

commented Jan 18, 2016

CL https://golang.org/cl/18708 mentions this issue.

@clnperez

This comment has been minimized.

Copy link
Author

commented Jan 18, 2016

Thanks. It looks like that's the only spot in the docker code that's an issue, so I can submit a PR to them for this.

clnperez added a commit to clnperez/moby that referenced this issue Jan 18, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

clnperez added a commit to clnperez/moby that referenced this issue Jan 18, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

@gopherbot gopherbot closed this in 49234ee Jan 18, 2016

@cpuguy83

This comment has been minimized.

Copy link

commented Jan 19, 2016

<3 Thanks. Probably haven't seen it before b/c we haven't done a release with GOMAXPROCS set higher than 1.

lami02 added a commit to lami02/docker that referenced this issue Jan 19, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

tiborvass added a commit to tiborvass/docker that referenced this issue Jan 26, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

aditirajagopal pushed a commit to aditirajagopal/docker that referenced this issue Feb 8, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

jheiss added a commit to jheiss/docker that referenced this issue Mar 10, 2016

Move closeNotify to fix panic with newer golang
This is happening now due to improvements in net/http:
golang/go@99fb191

To test, change the go version in the Dockerfile:
-ENV GO_VERSION 1.5.3
+ENV GO_VERSION 1.6beta2

More info here: golang/go#14001

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>

@golang golang locked and limited conversation to collaborators Jan 18, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.