panic when pulling images fails

[mikekap]

 E0718 18:13:12.156032   11785 runtime.go:58] Recovered from panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:52
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:40
 /usr/local/go/src/runtime/asm_amd64.s:472
 /usr/local/go/src/runtime/panic.go:443
 /usr/local/go/src/runtime/panic.go:62
 /usr/local/go/src/runtime/sigpanic_unix.go:24
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/client/image_pull.go:36
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockertools/kube_docker_client.go:316
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockertools/instrumented_docker.go:131
 <autogenerated>:42
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockertools/docker.go:187
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockertools/docker.go:228
 <autogenerated>:29
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockertools/docker_manager.go:853
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/container/serialized_image_puller.go:139
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/container/serialized_image_puller.go:63
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87
 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49
 /usr/local/go/src/runtime/asm_amd64.s:1998

This happens right after an image pull fails (in my case, due to failed auth):

[docker] time="2016-07-18T18:17:32.338917789Z" level=error msg="Handler for POST /images/create returned error: Get https://XXXX/v2/package/manifests/label: unauthorized: authentication required"

This causes the pod to be forever stuck in "creating container" state.

[yujuhong]

@mikekap what version of kubelet is this?

@ronnielai do you mind taking a look since you are looking at cleaning up the image puller code? Thanks! Feel free to unassign yourself for whatever reason.

[Random-Liu]

This is because we never use and set the "privilege function".
When the authentication failed, docker client tried to call the privilege function, but panic because it is nil.
The new engine-api has fixed this docker/engine-api@dea108d, but in our repo, no.

This is an engine-api bug, and we should bump up the engine-api version to fix this. And I'm not sure whether this needs a cherry-pick, :P.

[yujuhong]

IMO, this affects kubelet's reliability and should be patched to 1.3.

[aveshagarwal]

@kubernetes/rh-cluster-infra

[mikekap]

@yujuhong I tried with 1.3.0 and 1.3.2

[dims]

@Random-Liu @mikekap @ronnielai This issue can be closed now right? (Since the #29144 is merged already)