dashboard and nginx-ingress-controller fail to connect apiserver after minikubeVM restarted

[mumoshu]

I occasionally encountered this issue after minikubeVM got down somehow(maybe after my MacBook got into standby mode?) and then I had manually restarted minikubeVM with minikube start.

After minikubeVM is restarted, all the previously running pods are automatically started but dashboard and nginx-ingress-controller.

I'm using minikube v0.5.0

Log output for each pod is:

dashboard

$ kubectl logs -f kubernetes-dashboard-vj9e5 --namespace kube-system
Starting HTTP server on port 9090
Creating API server client for https://10.0.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: the server has asked for the client to provide credentials

nginx-ingress-controller

$ kubectl logs -f nginx-ingress-controller-4180429371-u6h57 --namespace chatwork-development
I0720 06:53:47.586855       1 main.go:96] Using build: https://github.com/bprashanth/contrib.git - git-5ae36b5
F0720 06:54:17.611703       1 main.go:125] unexpected error getting runtime information: timed out waiting for the condition

As you can see in main.go:125, this is the error message thrown when nginx-ingress-controller timed out while querying apiserver for pods. See https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/main.go#L125

/var/log/localkube.out

localkube (which I believe apiserver is included) seems to be running.

docker@minikubeVM:~$ ps auxww | grep localkube
docker   18675  0.0  0.0   9756   888 pts/0    S+   07:33   0:00 grep localkube
root     22143  4.2 17.3 21609148 177152 ?     Sl   05:57   4:08 /usr/local/bin/localkube --generate-certs=false --logtostderr=true

Looking into /var/log/localkube.out shows that there seems to be authentication issues:

E0720 07:34:48.178638   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0720 07:34:49.551197   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0720 07:34:50.958426   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
I0720 07:34:51.371461   22143 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/configmap/nginx-template" (spec.Name: "nginx-template-volume") to pod "9eaae271-4bee-11e6-8472-fa73e24ab7eb" (UID: "9eaae271-4bee-11e6-8472-fa73e24ab7eb"). Volume is already mounted to pod, but remount was requested.
I0720 07:34:51.371539   22143 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/default-token-e42c9" (spec.Name: "default-token-e42c9") to pod "9eaae271-4bee-11e6-8472-fa73e24ab7eb" (UID: "9eaae271-4bee-11e6-8472-fa73e24ab7eb"). Volume is already mounted to pod, but remount was requested.
I0720 07:34:51.375187   22143 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/default-token-e42c9" (spec.Name: "default-token-e42c9") pod "9eaae271-4bee-11e6-8472-fa73e24ab7eb" (UID: "9eaae271-4bee-11e6-8472-fa73e24ab7eb").
I0720 07:34:51.375436   22143 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/configmap/nginx-template" (spec.Name: "nginx-template-volume") pod "9eaae271-4bee-11e6-8472-fa73e24ab7eb" (UID: "9eaae271-4bee-11e6-8472-fa73e24ab7eb").
I0720 07:34:51.665611   22143 docker_manager.go:2279] checking backoff for container "nginx-ingress-lb" in pod "nginx-ingress-controller-4180429371-u6h57"
I0720 07:34:51.665852   22143 docker_manager.go:2293] Back-off 5m0s restarting failed container=nginx-ingress-lb pod=nginx-ingress-controller-4180429371-u6h57_chatwork-development(9eaae271-4bee-11e6-8472-fa73e24ab7eb)
E0720 07:34:51.665903   22143 pod_workers.go:183] Error syncing pod 9eaae271-4bee-11e6-8472-fa73e24ab7eb, skipping: failed to "StartContainer" for "nginx-ingress-lb" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=nginx-ingress-lb pod=nginx-ingress-controller-4180429371-u6h57_chatwork-development(9eaae271-4bee-11e6-8472-fa73e24ab7eb)"
E0720 07:34:52.327699   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0720 07:34:53.720040   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0720 07:34:55.197427   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
I0720 07:34:56.414783   22143 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/default-token-ms6p2" (spec.Name: "default-token-ms6p2") to pod "331329b3-4736-11e6-95bf-02433ea3fa41" (UID: "331329b3-4736-11e6-95bf-02433ea3fa41"). Volume is already mounted to pod, but remount was requested.
I0720 07:34:56.416223   22143 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/default-token-ms6p2" (spec.Name: "default-token-ms6p2") pod "331329b3-4736-11e6-95bf-02433ea3fa41" (UID: "331329b3-4736-11e6-95bf-02433ea3fa41").
E0720 07:34:56.566249   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
I0720 07:34:56.664697   22143 docker_manager.go:2279] checking backoff for container "kubernetes-dashboard" in pod "kubernetes-dashboard-vj9e5"
I0720 07:34:56.664764   22143 docker_manager.go:2293] Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-vj9e5_kube-system(331329b3-4736-11e6-95bf-02433ea3fa41)
E0720 07:34:56.664788   22143 pod_workers.go:183] Error syncing pod 331329b3-4736-11e6-95bf-02433ea3fa41, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-vj9e5_kube-system(331329b3-4736-11e6-95bf-02433ea3fa41)"
E0720 07:34:58.011258   22143 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
I0720 07:34:58.435740   22143 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/secret/default-token-e42c9" (spec.Name: "default-token-e42c9") to pod "24499ba7-4738-11e6-95bf-02433ea3fa41" (UID: "24499ba7-4738-11e6-95bf-02433ea3fa41"). Volume is already mounted to pod, but remount was requested.
I0720 07:34:58.437522   22143 operation_executor.go:720] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/default-token-e42c9" (spec.Name: "default-token-e42c9") pod "24499ba7-4738-11e6-95bf-02433ea3fa41" (UID: "24499ba7-4738-11e6-95bf-02433ea3fa41").

[jimmidyson]

Can you try with 0.6.0 please? We fixed some cert generation stuff that should fix this issue.

[dlorenc]

Hey,

This should have been fixed in #320, which went out in 0.6. Please re-open if it still repros for you there!

[dhilipsiva]

@dlorenc This is happening again. I tried a couple of time. First time installs works good. But after restart, it says: pod (nginx-ingress-controller-0f97j) failed to fit in any node fit failure summary on nodes : PodFitsHostPorts (1)

[chuckbutler]

@dhilipsiva - just trying to nail down some basics, that response message tells me there's another workload (pod) running on the host being scheduled that has occupied port 80/443. Can you confirm there are no other workloads running on the unit occupying that port as a NodePort binding, or launched at the system level?

[dhilipsiva]

@chuckbutler Apologies on the very late reply. Have been out of town. I actually, formatted my laptop. So you can ignore this for now, should I come across this error again, I will be sure to post the details here. Thank you for your response 😄