Heapster pod won't start on ACS cluster provisioned with Kubernetes version 1.6.2

[lachie83]

Details

Heapster pod won't start on ACS cluster provisioned with Kubernetes version 1.6.2

$ kubectl get pods -n kube-system
heapster-v1.2.0-559699904-ksjfs                 1/2       rpc error: code = 2 desc = failed to start container "2bb82c7425c1456ea78a4c743919173030ed9583cbfeaf66f76206237b6d036c": Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"/heapster\\\\\\\": stat /heapster: no such file or directory\\\"\\n\""}   3          32s

Fix

Single Master

Prerequisites

• Master FQDN - Found under the cluster in the Azure Portal
• SSH key used to provision the cluster

Steps

• Update Heapster addon deployment on master node

  ssh azureuzer@<master-fqdn> sudo sed -i 's/exechealthz-amd64:1.2/heapster:v1.2.0/' /etc/kubernetes/addons/kube-heapster-deployment.yaml

• Using kubectl (either locally or on the master) delete the current Heapster deployment

  kubectl delete deploy heapster-v1.2.0 -n kube-system

• Watch pods in kube-system namespace and wait for addon-manager to recreate the Heapster pods

$ kubectl get pods -n kube-system -w
heapster-v1.2.0-1516787090-cp68p   0/2       Pending   0         0s
heapster-v1.2.0-1516787090-cp68p   0/2       Pending   0         0s
heapster-v1.2.0-1516787090-cp68p   0/2       ContainerCreating   0         1s
heapster-v1.2.0-1516787090-cp68p   2/2       Running   0         3s

Multi-master

Prerequisites

• Master FQDN - Found under the cluster in the Azure Portal
• SSH key used to provision the cluster

Steps

• Load the provisioned ssh-key into the local ssh-agent (this procedure will differ based on the ssh-client)

ssh-add <key-path>
ssh-add -l

• Update the Heapster deployment manifest on all master nodes

for master in `kubectl get nodes -l role=master -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`; \
    do echo $i; \
    ssh -A azureuzer@<master-fqdn> ssh $master -oStrictHostKeyChecking=no sudo sed -i 's/exechealthz-amd64:1.2/heapster:v1.2.0/' /etc/kubernetes/addons/kube-heapster-deployment.yaml; \
    done

• Using kubectl (either locally or on the master) delete the current Heapster deployment

  kubectl delete deploy heapster-v1.2.0 -n kube-system

• Watch pods in kube-system namespace and wait for addon-manager to recreate the Heapster pods

$ kubectl get pods -n kube-system -w
heapster-v1.2.0-1516787090-cp68p   0/2       Pending   0         0s
heapster-v1.2.0-1516787090-cp68p   0/2       Pending   0         0s
heapster-v1.2.0-1516787090-cp68p   0/2       ContainerCreating   0         1s
heapster-v1.2.0-1516787090-cp68p   2/2       Running   0         3s

[Starefossen]

Thanks for sharing this resolution 👍🏼😄

[anhowe]

this is now rolled out everywhere

[Starefossen]

@anhowe can I update my ACS Kubernetes cluster automatically other than the manual fix described by @lachie83?

[marceldempers]

Thanks for this solution - We experienced the same issue, this fixed it! 👍