Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker daemon panic in startContainer.go when passing wrong hostConfig #32136

Closed
grexe opened this issue Mar 27, 2017 · 10 comments
Closed

docker daemon panic in startContainer.go when passing wrong hostConfig #32136

grexe opened this issue Mar 27, 2017 · 10 comments
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.12

Comments

@grexe
Copy link

grexe commented Mar 27, 2017

When passing wrong parameters to extraHosts as part of the HostConfig, the error is propagated all the way down to the startContainer go code, and causes an index out of range error there.
The daemon should better check arguments and report a suitable HTTP error response instead.

I was using a Java docker client library (8.1.3-SNAPSHOT)
and docker version 1.12.6, build 78d1802.
The lib just sends out the REST call to the local docker daemon over HTTP in the end.

Steps to reproduce the issue:

  1. call your local docker daemon with the usual REST URI for createContainer and a HostConfig that contains a newline or single argument instead of the expected "host:alias" format
  2. call startContainer
  3. you get a stack trace from the docker daemon indicating an index out of range error in the go code for startContainer (I guess it expects 2 arguments but only gets 1 and so the access fails)

Describe the results you received:
The container is created as normal and the call returns a container ID (it should already fail because of the wrong argument for hostConfig).
Only when you try start this configured container, it crashes the daemon.

Describe the results you expected:
For a wrong configuration, I'd expect the createContainer method to check the arguments are valid, and already report an error if they are invalid, instead of returning an ID, i.e. fail fast.

Other info
I am using a private repo and the ZFS backend, but this should not matter, also could reproduce with standard containers from the global registry.

Output of docker info:

Containers: 340
 Running: 0
 Paused: 0
 Stopped: 340
Images: 6146
Server Version: 1.12.6
Storage Driver: zfs
 Zpool: fpool
 Zpool Health: ONLINE
 Parent Dataset: fpool/docker
 Space Used By Parent: 85179502592
 Space Available: 282574352384
 Parent Quota: no
 Compression: lz4
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.8.0-42-generic
Operating System: Ubuntu 16.10
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.29 GiB
Name: gregor-Aspire-VN7-592G
ID: MUSL:FDKF:NTZA:QIHD:SCGL:G2YZ:QZ7E:32DY:AV4B:WXPS:WQVF:BERD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false                                                                                                   
Debug Mode (server): false                                                                                                   
Registry: https://index.docker.io/v1/                                                                                        
WARNING: No swap limit support                                                                                               
Insecure Registries:                                                                                                         
 127.0.0.0/8
@vdemeester vdemeester added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.12 labels Mar 27, 2017
@cpuguy83
Copy link
Member

Can you paste the stack from the panic?

@thaJeztah
Copy link
Member

Would this PR resolve the validation part? #31384

@grexe
Copy link
Author

grexe commented Mar 27, 2017

@thaJeztah I was just reproducing to get the stack:)

dockerd[8394]: http: panic serving @: runtime error: index out of range
                                                       goroutine 1290 [running]:
                                                       net/http.(*conn).serve.func1(0xc821f81300)
                                                               /usr/local/go/src/net/http/server.go:1389 +0xc1
                                                       panic(0x1a9f880, 0xc82000e040)
                                                               /usr/local/go/src/runtime/panic.go:443 +0x4e9
                                                       github.com/docker/docker/daemon.(*Daemon).buildSandboxOptions(0xc82047a1a0, 0xc821215680, 0x0, 0x0, 0x0, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:121 +0x389b
                                                       github.com/docker/docker/daemon.(*Daemon).connectToNetwork(0xc82047a1a0, 0xc821215680, 0x1d81150, 0x6, 0xc821f16e40, 0x4da800, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:590 +0x598
                                                       github.com/docker/docker/daemon.(*Daemon).allocateNetwork(0xc82047a1a0, 0xc821215680, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:417 +0x39b
                                                       github.com/docker/docker/daemon.(*Daemon).initializeNetworking(0xc82047a1a0, 0xc821215680, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/container_operations.go:703 +0x42a
                                                       github.com/docker/docker/daemon.(*Daemon).containerStart(0xc82047a1a0, 0xc821215680, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/start.go:126 +0x1fe
                                                       github.com/docker/docker/daemon.(*Daemon).ContainerStart(0xc82047a1a0, 0xc820c313f1, 0x40, 0x0, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/daemon/start.go:80 +0x5dd
                                                       github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStart(0xc820f058c0, 0x7fe76f360140, 0xc82a04c300, 0x7fe76c162818, 0xc820357110, 0xc82a284d20, 0xc82a04c270, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/router/container/container_routes.go:154 +0x145
                                                       github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/docker/api/server/router/container.postContainersStart)-fm(0x7fe76f360140, 0xc82a04c300, 0x7fe76c162818, 0xc820357110, 0xc82a284d20, 0xc82a04c270, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/router/container/container.go:60 +0x74
                                                       github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x7fe76f360140, 0xc82a04c300, 0x7fe76c162818, 0xc820357110, 0xc82a284d20, 0xc82a04c270, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:56 +0x9aa
                                                       github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x7fe76f360140, 0xc82a04c2d0, 0x7fe76c162818, 0xc820357110, 0xc82a284d20, 0xc82a04c270, 0x0, 0x0)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/middleware/user_agent.go:45 +0x68e
                                                       github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7fe76c162818, 0xc820357110, 0xc82a284d20)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/server.go:139 +0x138
                                                       net/http.HandlerFunc.ServeHTTP(0xc820c7a9c0, 0x7fe76c162818, 0xc820357110, 0xc82a284d20)
                                                               /usr/local/go/src/net/http/server.go:1618 +0x3a
                                                       github.com/gorilla/mux.(*Router).ServeHTTP(0xc820d66230, 0x7fe76c162818, 0xc820357110, 0xc82a284d20)
                                                               /usr/src/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
                                                       github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc8213cfe80, 0x7fe76c162818, 0xc820357110, 0xc82a284d20)
                                                               /usr/src/docker/.gopath/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
                                                       net/http.serverHandler.ServeHTTP(0xc8203e2400, 0x7fe76c162818, 0xc820357110, 0xc82a284d20)
                                                               /usr/local/go/src/net/http/server.go:2081 +0x19e
                                                       net/http.(*conn).serve(0xc821f81300)
                                                               /usr/local/go/src/net/http/server.go:1472 +0xf2e
                                                       created by net/http.(*Server).Serve
                                                               /usr/local/go/src/net/http/server.go:2137 +0x44e

@grexe
Copy link
Author

grexe commented Mar 27, 2017

@thaJeztah I see your PR touches exactly this area, but didn't see the implementation of validateExtraHosts checks, and my Go knowledge is still quite rusty (hehe;)

@thaJeztah
Copy link
Member

The ValidateExtraHost is now performed here; https://github.com/docker/docker/blob/6129e6ce3ebd1bd6c9db76e196248c613fda6e40/opts/hosts.go#L153-L165, and ValidateIPAddress; https://github.com/docker/docker/blob/83e29a5d87e3c1a0d4a9ae0d6dfe14a9ceb9bb20/opts/opts.go#L227-L234

Would you be able to provide a minimal example of the container's config that could be used to reproduce the issue? (e.g. the literal JSON data that's sent?)

@grexe
Copy link
Author

grexe commented Mar 27, 2017

@thaJeztah ok, just not so straightforward to dump the request JSON right now, I'll have to add a LoggingFilter to see...

@thaJeztah
Copy link
Member

@grexe if you put the docker daemon in debug mode, then API requests are included in the logs. Easiest way is to create a /etc/docker/daemon.json configuration file (as described here in the docs; https://docs.docker.com/engine/reference/commandline/dockerd/#linux-configuration-file), containing;

{"debug":true}

After that restart or "reload" the daemon;

systemctl reload docker

@grexe
Copy link
Author

grexe commented Mar 27, 2017

thanks @thaJeztah but that actually only prints out the request/response, but not the body.
Here's the full output with Jersey's handy LoggingFilter:

create container request:
POST unix://localhost:80/containers/create?name=a4e7ae1c2f73df0e-118338e5a182c15d
{"Cmd":["sh","-c","cat /etc/hosts | grep extrahost"],"Image":"busybox:latest","Volumes":{},"HostConfig":{"ExtraHosts":["extrahost-1.2.3.4"]}}

response:
201
Content-Length: 90
Content-Type: application/json
Date: Mon, 27 Mar 2017 15:12:57 GMT
Server: Docker/1.12.6 (linux)
{"Id":"66943d238d9e72556f247d6b0152f03d406870d5f76dae5bb3e9af54997d7a5e","Warnings":null}

start container request:
POST unix://localhost:80/containers/66943d238d9e72556f247d6b0152f03d406870d5f76dae5bb3e9af54997d7a5e/start

response:
I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->unix://localhost:80: The target server failed to respond
500
Content-Length: 90
Content-Type: application/json
Date: Mon, 27 Mar 2017 15:12:57 GMT
Server: Docker/1.12.6 (linux)
{"message":"service endpoint with name a4e7ae1c2f73df0e-118338e5a182c15d already exists"}

(I think the "service endpoint already exists" is new, but the actual error stays the same).

@thaJeztah
Copy link
Member

that actually only prints out the request/response, but not the body.

Hm, interesting, I'll have to check that 🤔

Thanks for the reproduction steps, I'll give it a go later with the 17.04 version to check if it's resolved 👍

@thaJeztah
Copy link
Member

Sorry for the delay; I just tested this on Docker 17.06 (it's what I have installed now 😄);

curl -v \
  --unix-socket /var/run/docker.sock \
  "http://localhost/containers/create?name=a4e7ae1c2f73df0e-118338e5a182c15d" \
  -H "Content-Type: application/json" \
  -d '{"Cmd":["sh","-c","cat /etc/hosts | grep extrahost"],"Image":"busybox:latest","Volumes":{},"HostConfig":{"ExtraHosts":["extrahost-1.2.3.4"]}}'

Which produced this response;

*   Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> POST /containers/create?name=a4e7ae1c2f73df0e-118338e5a182c15d HTTP/1.1
> Host: localhost
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 141
>
* upload completely sent off: 141 out of 141 bytes
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.30
< Content-Length: 61
< Content-Type: application/json
< Date: Mon, 12 Jun 2017 22:12:18 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.06.0-ce-rc2 (linux)
<
{"message":"bad format for add-host: \"extrahost-1.2.3.4\""}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact

After that, no container was created, and creating a container without the invalid ExtraHosts worked (no "service endpoint .. already exists");

curl -v \
  --unix-socket /var/run/docker.sock \
  "http://localhost/containers/create?name=a4e7ae1c2f73df0e-118338e5a182c15d" \
  -H "Content-Type: application/json" \
  -d '{"Cmd":["sh","-c","cat /etc/hosts | grep extrahost"],"Image":"busybox:latest","Volumes":{}}'
*   Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> POST /containers/create?name=a4e7ae1c2f73df0e-118338e5a182c15d HTTP/1.1
> Host: localhost
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 91
>
* upload completely sent off: 91 out of 91 bytes
< HTTP/1.1 201 Created
< Api-Version: 1.30
< Content-Length: 90
< Content-Type: application/json
< Date: Mon, 12 Jun 2017 22:15:02 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.06.0-ce-rc2 (linux)
<
{"Id":"e27dc2a6d456517c419c30e5b1e4e0097a78d0d8e324561d70feee6d1c8d4f64","Warnings":null}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact

I'll close this issue, because this looks to be resolved, but let me know if there's still something you think should be addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. version/1.12
Projects
None yet
Development

No branches or pull requests

4 participants