docker daemon receives SIGSEGV

[xylnao]

Description of problem:

A docker daemon receives a signal SIGSEGV during creation of a container.

docker version:

Client version: 1.8.0-dev
Client API version: 1.20
Go version (client): go1.4.2
Git commit (client): cb6ca19
OS/Arch (client): linux/amd64
Server version: 1.8.0-dev
Server API version: 1.20
Go version (server): go1.4.2
Git commit (server): cb6ca19
OS/Arch (server): linux/amd64

docker info:

Containers: 26
Images: 143
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 195
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-30-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 986.8 MiB
Name: ubuntu
ID: J7AL:63BJ:74EP:LB3B:GGMI:SPOK:334A:FJKY:XSG4:N5OY:YQLS:OXAE
WARNING: No swap limit support

uname -a:

Linux ubuntu 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

VMware Professional Version 5.0.5 (1945692)

How reproducible:

Always

Steps to Reproduce:

1. Start a docker daemon from a gdb.

% gdb docker
run -d

Run any small command inside a docker container from another shell prompt.

% docker run --rm ubuntu echo hello

Actual Results:

The docker daemon receives a signal SIGSEGV.

Expected Results:

The request is processed normally.

Additional info:

If you continue the debug session after this SIGSEGV, new threads are generated and the request looks processed normally.

% gdb bundles/1.8.0-dev/binary/docker-1.8.0-dev 
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bundles/1.8.0-dev/binary/docker-1.8.0-dev...done.
(gdb) run -d
Starting program: /home/nao/docker/bundles/1.8.0-dev/binary/docker-1.8.0-dev -d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7e4a700 (LWP 12883)]
[New Thread 0x7ffff7649700 (LWP 12884)]
[New Thread 0x7ffff6e48700 (LWP 12885)]
[New Thread 0x7ffff6647700 (LWP 12886)]
[New Thread 0x7ffff5e46700 (LWP 12887)]
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
INFO[0000] [graphdriver] using prior storage driver "aufs" 
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1 
WARN[0000] Your kernel does not support swap memory limit. 
INFO[0000] Loading containers: start.                   
....................................................
INFO[0000] Loading containers: done.                    
INFO[0000] Daemon has completed initialization          
INFO[0000] Docker daemon                                 commit=cb6ca19 execdriver=native-0.2 graphdriver=aufs version=1.8.0-dev
INFO[0004] POST /v1.20/containers/create                
INFO[0004] POST /v1.20/containers/52c6d7846861ff018deb45dea854ee0f1a93133baabad48ac04d42e6aedfb7bb/attach?stderr=1&stdout=1&stream=1 
INFO[0004] POST /v1.20/containers/52c6d7846861ff018deb45dea854ee0f1a93133baabad48ac04d42e6aedfb7bb/start 
INFO[0004] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4] 

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6647700 (LWP 12886)]
net.networkNumberAndMask (n=0x0, ip=..., m=...) at /usr/local/go/src/net/ip.go:436
436     if ip = n.IP.To4(); ip == nil {
(gdb) bt
#0  net.networkNumberAndMask (n=0x0, ip=..., m=...) at /usr/local/go/src/net/ip.go:436
#1  0x000000000063370b in net.(*IPNet).String (n=0x0, ~r0=...)
    at /usr/local/go/src/net/ip.go:486
#2  0x0000000000482f59 in fmt.(*pp).handleMethods (p=0xc2080d2750, verb=113, depth=0, 
    handled=true) at /usr/local/go/src/fmt/print.go:720
#3  0x00000000004833e8 in fmt.(*pp).printArg (p=0xc2080d2750, arg=..., verb=113, depth=0, 
    wasString=false) at /usr/local/go/src/fmt/print.go:794
#4  0x000000000048b23d in fmt.(*pp).doPrintf (p=..., format=..., a=...)
    at /usr/local/go/src/fmt/print.go:1183
#5  0x000000000047de28 in fmt.Sprintf (format=..., a=..., ~r2=...)
    at /usr/local/go/src/fmt/print.go:203
#6  0x00000000009392cb in github.com/docker/libnetwork/sandbox.configureInterface (
    iface=..., i=0xc208030380, ~r2=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go:258
#7  0x000000000093efd6 in github.com/docker/libnetwork/sandbox.func·004 (callerFD=12, 
    ~r1=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go:233
#8  0x000000000093bb57 in github.com/docker/libnetwork/sandbox.nsInvoke (path=..., prefunc=
    {void (int, error *)} 0xc2082c8cf8, postfunc={void (int, error *)} 0xc2082c8d00, 
    ~r3=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox/namespace_linux.go:275
#9  0x0000000000938d69 in github.com/docker/libnetwork/sandbox.(*networkNamespace).AddInterface (n=0xc2081c9180, srcName=..., dstPrefix=..., options=..., ~r3=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go:247
#10 0x00000000007b05b4 in github.com/docker/libnetwork.(*sandboxData).addEndpoint (
    s=0xc2082ddcc0, ep=0xc20827cbd0, ~r1=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandboxdata.go:92
#11 0x00000000007b17a3 in github.com/docker/libnetwork.(*controller).sandboxAdd (
    c=0xc208242e80, key=..., create=true, ep=0xc20827cbd0, ~r3=..., ~r4=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandboxdata.go:198
#12 0x00000000007a32a8 in github.com/docker/libnetwork.(*endpoint).Join (ep=0xc20827cbd0, 
    containerID=..., options=..., ~r2=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/endpoint.go:424
#13 0x00000000004f5877 in github.com/docker/docker/daemon.(*Container).configureNetwork (
    container=0xc2082ba1e0, networkName=..., service=..., networkDriver=..., 
    canCreateNetwork=true, ~r4=...)
    at /go/src/github.com/docker/docker/daemon/container_linux.go:867
#14 0x00000000004f523c in github.com/docker/docker/daemon.(*Container).AllocateNetwork (
    container=0xc2082ba1e0, ~r0=...)
    at /go/src/github.com/docker/docker/daemon/container_linux.go:821
#15 0x00000000004f5f7c in github.com/docker/docker/daemon.(*Container).initializeNetworking
    (container=0xc2082ba1e0, ~r0=...)
    at /go/src/github.com/docker/docker/daemon/container_linux.go:918
#16 0x00000000004e229f in github.com/docker/docker/daemon.(*Container).Start (
    container=0xc2082ba1e0, err=...)
    at /go/src/github.com/docker/docker/daemon/container.go:255
#17 0x000000000051806c in github.com/docker/docker/daemon.(*Daemon).ContainerStart (
    daemon=0xc2080616c0, name=..., hostConfig=0x0, ~r2=...)
    at /go/src/github.com/docker/docker/daemon/start.go:35
#18 0x00000000004d0f42 in github.com/docker/docker/api/server.(*Server).postContainersStart
    (s=0xc20808ad00, version=..., w=..., r=0xc2080d2410, vars=0xc208301a70, ~r4=...)
    at /go/src/github.com/docker/docker/api/server/server.go:1031
#19 0x00000000004ddb3b in github.com/docker/docker/api/server.*Server.(github.com/docker/docker/api/server.postContainersStart)·fm (a0=..., a1=..., a2=0xc2080d2410, a3=0xc208301a70, 
    r0=...) at /go/src/github.com/docker/docker/api/server/server.go:1544
#20 0x00000000004dc565 in github.com/docker/docker/api/server.func·008 (w=..., 
    r=0xc2080d2410) at /go/src/github.com/docker/docker/api/server/server.go:1495
#21 0x00000000006d50e1 in net/http.HandlerFunc.ServeHTTP (f=
    {void (net/http.ResponseWriter, struct net/http.Request *)} 0xc2082c9ba8, w=..., 
    r=0xc2080d2410) at /usr/local/go/src/net/http/server.go:1265
#22 0x00000000006f3ea7 in github.com/gorilla/mux.(*Router).ServeHTTP (r=0xc20810ba90, w=..., req=0xc2080d2410)
    at /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98
#23 0x00000000006d721a in net/http.serverHandler.ServeHTTP (sh=..., rw=..., 
    req=0xc2080d2410) at /usr/local/go/src/net/http/server.go:1703
#24 0x00000000006d4c37 in net/http.(*conn).serve (c=0xc2081c92c0)
    at /usr/local/go/src/net/http/server.go:1204
#25 0x000000000044c051 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:2232
#26 0x000000c2081c92c0 in ?? ()
#27 0x3664313437343230 in ?? ()
#28 0x3734393935323063 in ?? ()
#29 0x6c2f646461306665 in ?? ()
#30 0x0000000072657961 in ?? ()
#31 0x45474155474e414c in ?? ()
#32 0x000053555f6e653d in ?? ()
#33 0x3d454d414e474f4c in ?? ()
#34 0x00000000746f6f72 in ?? ()
#35 0x3d59414c50534944 in ?? ()
#36 0x00000000302e303a in ?? ()
#37 0x4449475f4f445553 in ?? ()
#38 0x000000303030313d in ?? ()
#39 0x0000000000000026 in ?? ()
#40 0x0000000000000000 in ?? ()
(gdb) frame 6
#6  0x00000000009392cb in github.com/docker/libnetwork/sandbox.configureInterface (
    iface=..., i=0xc208030380, ~r2=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go:258
258         {setInterfaceIPv6, fmt.Sprintf("error setting interface %q IPv6 to %q", ifaceName, i.AddressIPv6())},
(gdb) list
253         Fn         func(netlink.Link, *nwIface) error
254         ErrMessage string
255     }{
256         {setInterfaceName, fmt.Sprintf("error renaming interface %q to %q", ifaceName, i.DstName())},
257         {setInterfaceIP, fmt.Sprintf("error setting interface %q IP to %q", ifaceName, i.Address())},
258         {setInterfaceIPv6, fmt.Sprintf("error setting interface %q IPv6 to %q", ifaceName, i.AddressIPv6())},
259         {setInterfaceRoutes, fmt.Sprintf("error setting interface %q routes to %q", ifaceName, i.Routes())},
260         {setInterfaceMaster, fmt.Sprintf("error setting interface %q master to %q", ifaceName, i.DstMaster())},
261     }
262 
(gdb) p ip
No symbol "ip" in current context.
(gdb) p i
$1 = (struct github.com/docker/libnetwork/sandbox.nwIface *) 0xc208030380
(gdb) p *i
$2 = {srcName = 0xc2082ed4b0 "veth28eae8e", dstName = 0xc2082ca0a0 "eth0", 
  master = 0x0 "", dstMaster = 0x0 "", address = 0xc208276aa0, addressIPv6 = 0x0, 
  routes = {array = 0x0, len = 0, cap = 0}, bridge = false, ns = 0xc2081c9180, 
  sync.Mutex = {state = 0, sema = 0}}
(gdb) c
Continuing.
[New Thread 0x7ffff5645700 (LWP 12962)]
[New Thread 0x7ffff4e44700 (LWP 12963)]
[New Thread 0x7fffdffff700 (LWP 12964)]
INFO[0052] POST /v1.20/containers/52c6d7846861ff018deb45dea854ee0f1a93133baabad48ac04d42e6aedfb7bb/wait 
INFO[0052] GET /v1.20/containers/52c6d7846861ff018deb45dea854ee0f1a93133baabad48ac04d42e6aedfb7bb/json 
INFO[0052] DELETE /v1.20/containers/52c6d7846861ff018deb45dea854ee0f1a93133baabad48ac04d42e6aedfb7bb?v=1

A small patch seems to fix this problem, though I am not sure docker people like this idea. Sould I proceed to PR?

diff --git a/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go b/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go
index a89b3e7..71ec064 100644
--- a/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go
+++ b/vendor/src/github.com/docker/libnetwork/sandbox/interface_linux.go
@@ -249,22 +249,37 @@ func (n *networkNamespace) AddInterface(srcName, dstPrefix string, options ...If

 func configureInterface(iface netlink.Link, i *nwIface) error {
    ifaceName := iface.Attrs().Name
-   ifaceConfigurators := []struct {
-       Fn         func(netlink.Link, *nwIface) error
-       ErrMessage string
-   }{
-       {setInterfaceName, fmt.Sprintf("error renaming interface %q to %q", ifaceName, i.DstName())},
-       {setInterfaceIP, fmt.Sprintf("error setting interface %q IP to %q", ifaceName, i.Address())},
-       {setInterfaceIPv6, fmt.Sprintf("error setting interface %q IPv6 to %q", ifaceName, i.AddressIPv6())},
-       {setInterfaceRoutes, fmt.Sprintf("error setting interface %q routes to %q", ifaceName, i.Routes())},
-       {setInterfaceMaster, fmt.Sprintf("error setting interface %q master to %q", ifaceName, i.DstMaster())},
+   if i.DstName() != "" {
+       err := setInterfaceName(iface, i)
+       if err != nil {
+           return fmt.Errorf(fmt.Sprintf("error renaming interface %q to %q: %v", ifaceName, i.DstName(), err))
+       }
    }
-
-   for _, config := range ifaceConfigurators {
-       if err := config.Fn(iface, i); err != nil {
-           return fmt.Errorf("%s: %v", config.ErrMessage, err)
+   if i.Address() != nil {
+       err := setInterfaceIP(iface, i)
+       if err != nil {
+           return fmt.Errorf(fmt.Sprintf("error setting interface %q IP to %q: %v", ifaceName, i.Address(), err))
+       }
+   }
+   if i.AddressIPv6() != nil {
+       err := setInterfaceIPv6(iface, i)
+       if err != nil {
+           return fmt.Errorf(fmt.Sprintf("error setting interface %q IPv6 to %q: %v", ifaceName, i.AddressIPv6(), err))
        }
    }
+   if i.Routes() != nil {
+       err := setInterfaceRoutes(iface, i)
+       if err != nil {
+           return fmt.Errorf(fmt.Sprintf("error setting interface %q routes to %q: %v", ifaceName, i.Routes(), err))
+       }
+   }
+   if i.DstMaster() != "" {
+       err := setInterfaceMaster(iface, i)
+       if err != nil {
+           return fmt.Errorf(fmt.Sprintf("error setting interface %q master to %q: %v", ifaceName, i.DstMaster(), err))
+       }
+   }
+
    return nil
 }

[cyphar]

Since the patch applies to a vendored package (in particular libnetwork -- docker/libnetwork), I'd make the patch against that tree and send a PR there. Once it's been merged, propose a PR here that updates the vendored version to include your fix.

On a side note, can this segfault be caused without being run under GDB?

[xylnao]

Cyphar, thank you for your comment.

I made a minimum program, say nw.go, to reproduce this strange behavior:

package main

import(
    "fmt"
    "net"
)

func main() {
    fmt.Printf("nil: %q\n", (*net.IPNet)(nil));
}

This program receives a signal SIGSEGV only if it is invoked from a gdb. This behavior must be something to do with the Go runtime or the Linux kernel itself.

$ go build nw.go
$ ./nw
nil: <nil>
$ gdb nw
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from nw...done.
Loading Go Runtime support.
(gdb) run
Starting program: /home/nao/work/nw 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff77f6700 (LWP 4674)]
[New Thread 0x7ffff6ff5700 (LWP 4675)]
[New Thread 0x7ffff67f4700 (LWP 4676)]

Program received signal SIGSEGV, Segmentation fault.
0x0000000000472222 in net.networkNumberAndMask (n=0x0, ip=..., m=...)
    at /home/nao/go/src/net/ip.go:434
434     if ip = n.IP.To4(); ip == nil {
(gdb) bt
#0  0x0000000000472222 in net.networkNumberAndMask (n=0x0, ip=..., m=...)
    at /home/nao/go/src/net/ip.go:434
#1  0x0000000000472572 in net.(*IPNet).String (n=0x0, ~r0="")
    at /home/nao/go/src/net/ip.go:484
#2  0x000000000045cfbb in fmt.(*pp).handleMethods (p=0xc20806c8f0, verb=113, 
    depth=0, handled=true) at /home/nao/go/src/fmt/print.go:725
#3  0x000000000045d537 in fmt.(*pp).printArg (p=0xc20806c8f0, arg=..., 
    verb=113, depth=0, wasString=false) at /home/nao/go/src/fmt/print.go:801
#4  0x000000000046462d in fmt.(*pp).doPrintf (p= []uint8, format="nil: %q\n", 
    a= []interface {} = {...}) at /home/nao/go/src/fmt/print.go:1197
#5  0x00000000004586b4 in fmt.Fprintf (w=..., format="nil: %q\n", 
    a= []interface {} = {...}, n=4244826, err=...)
    at /home/nao/go/src/fmt/print.go:188
#6  0x00000000004587d4 in fmt.Printf (format="nil: %q\n", 
    a= []interface {} = {...}, n=0, err=...)
    at /home/nao/go/src/fmt/print.go:197
#7  0x0000000000400c9e in main.main () at /home/nao/work/nw.go:9
(gdb) p n
$1 = (struct net.IPNet *) 0x0
(gdb) c
Continuing.
nil: <nil>
[Thread 0x7ffff67f4700 (LWP 4676) exited]
[Thread 0x7ffff6ff5700 (LWP 4675) exited]
[Thread 0x7ffff77f6700 (LWP 4674) exited]
[Inferior 1 (process 4670) exited normally]
(gdb) quit

[cyphar]

@xylnao I'd send this to the Go mailing lists, see what they have to say. I think it's probably a bug in the Go runtime.

[mrjana]

@xylnao @cyphar The fmt package handles all nil pointer references inside a fmt.*printf call to generate a <nil>string. It does this by actually referencing the pointer even if it is nil and let the kernel generate a SIGSEGV and use the panic/recover architecture to catch the signal and recover from it just in this case. The following is the relevant go fmt package code which actually does this:

func (p *pp) catchPanic(arg interface{}, verb rune) {
        if err := recover(); err != nil {
            // If it's a nil pointer, just say "<nil>". The likeliest causes are a
            // Stringer that fails to guard against nil or a nil pointer for a
            // value receiver, and in either case, "<nil>" is a nice result.
            if v := reflect.ValueOf(arg); v.Kind() == reflect.Ptr && v.IsNil() {
                p.buf.Write(nilAngleBytes)
                return

But when you invoke the same program from gdb, gdb traps SIGSEGV before the program which the go runtime is part of gets the signal and that is why you are seeing the SIGSEGV generated only inside gdb.

So this is not a bug but a feature of go runtime and fmt package and hence we should close this issue.

[cyphar]

@mrjana We can close the issue sure, but I feel like this deserves some discussion on the Go mailing list. It's weird that a Go binary doesn't play well with gdb.

[mrjana]

@cyphar I am not saying this should not be discussed in Go mailing list. But the way gdb works is that when you run a program inside gdb, the program is an inferior process of gdb and with ptrace gdb asks the kernel that all signals generated for the inferior process to be sent to gdb instead. This is how ptrace works and hence the the inferior processes doesn't get to have any say in this unless gdb decides to pass the signal to the process(You can set this up in gdb using the handle command).

So all in all, this is an expected behavior in both gdb and go runtime.

[icecrime]

I'm closing this issue per the conversation above, thanks all.