Docker 1.11.0 crash with invalid memory address or nil pointer dereference on libnetwork.(*resolver).forwardQueryStart

[PierreF]

---

BUG REPORT INFORMATION

Output of docker version:

Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:38:59 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:38:59 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 8
 Running: 2
 Paused: 0
 Stopped: 6
Images: 426
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 568
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.2.0-35-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.704 GiB
Name: ubuntu
ID: 2U4Z:HNRO:CIHL:JXJQ:CMMB:QN2D:WL5O:KK72:UGDQ:RRYQ:W2DH:N72B
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

I don't know how to reproduce it, but it already occured 2 times today (I upgraded to 1.11.0 today). Both time Docker crashed with the following error (in syslog):

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x30 pc=0x87d73d]
goroutine 177134 [running]:
github.com/docker/libnetwork.(*resolver).forwardQueryStart(0xc8215d2600, 0x0, 0x0, 0xc820c6ae10, 0x0)
/usr/src/docker/vendor/src/github.com/docker/libnetwork/resolver.go:442 +0x5d
github.com/docker/libnetwork.(*resolver).ServeDNS(0xc8215d2600, 0x0, 0x0, 0xc820c6ae10)
/usr/src/docker/vendor/src/github.com/docker/libnetwork/resolver.go:391 +0xde4
github.com/miekg/dns.(*Server).serve(0xc820b0f450, 0x7f9f18445530, 0xc820abc870, 0x7f9f1cab02f8, 0xc8215d2600, 0xc820f23a00, 0x39, 0x200, 0xc8215b05a8, 0xc82020dd00, ...)
/usr/src/docker/vendor/src/github.com/miekg/dns/server.go:535 +0x7c1
created by github.com/miekg/dns.(*Server).serveUDP
/usr/src/docker/vendor/src/github.com/miekg/dns/server.go:489 +0x3d5

[sanimej]

@PierreF Are you using multiple external resolvers (either through --dns or from the host's resolv.conf file) ?

[PierreF]

I don't specify --dns and host resolv.conf is

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search home

[sanimej]

127.x.x.x from host resolv.conf are not used within the container. So in this case by default Docker will use 8.8.8.8 and 8.8.4.4 as the external servers.

From a quick analysis of the crash decode looks like this can happen in certain cases with multiple external servers. Till this is fixed as a work around can you try passing one external server either though --dns option for the daemon (all containers will inherit that config) or for the individual containers's docker run through --dns ?

[PierreF]

I've added "--dns 8.8.8.8" to docker daemon command line, restarted everything and no issue for last 12 hours. Seems that the workaround fix this issue.

[mrgleam]

Me too.
I use this command
$ docker run -t -i --rm --dns 8.8.8.8 --net=backend nginx cat /etc/resolv.conf

output :
nameserver 127.0.0.11
options nods:0

and i try to use ping google.com in container
It work in first time but in second time, i can not ping anything and docker daemon crash.

if i add this in resolve.conf
nameserver 127.0.0.11
nameserver 8.8.8.8
options nods:0

It work for me.

but how to add second name server? --dns not work.

[mrgleam]

Ok i use docker-compose

I add

command: sh -c "echo "nameserver 8.8.8.8" >> /etc/resolv.conf && [your foreground command]"

It work for me.

[icecrime]

@mavenugo Can you please let us know if you have a fix for this one, or if you're confident you'll have one in the 1.11.1 patch release timeframe? Thanks!

[mavenugo]

@icecrime yes. this is already resolved in libnetwork via moby/libnetwork#1125 and we will vendor it in shortly.

[thaJeztah]

This will be fixed in 1.11.1 through #22261 (which was just merged)