Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard not working after upgrade to 1.14.8 #1452

Closed
markmcgookin opened this issue Feb 17, 2020 · 7 comments
Closed

Dashboard not working after upgrade to 1.14.8 #1452

markmcgookin opened this issue Feb 17, 2020 · 7 comments
Labels

Comments

@markmcgookin
Copy link

What happened:
Upgraded Azure AKS from 1.14.3 to 1.14.8 using the steps

az aks get-upgrades --resource-group myResourceGroup --name myAKSCluster --output table
az aks upgrade --resource-group myResourceGroup --name myAKSCluster --kubernetes-version 1.14.8

Now when trying to go to the dashboard we just get a list of API paths...


{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/",
"/apis/admissionregistration.k8s.io",

If I run

kubectl get pods -n kube-system
I can see the dashboard pod running. But if I look into the logs for that pod with

kubectl logs kubernetes-dashboard-<our-pod-id> -n kube-system

I get lots of repeats of this error

2020/02/17 10:19:43 Metric client health check failed: unknown (get services heapster). Retrying in 30 seconds.

I've checked that there is a clusterrolebinding for cluster-admin for the service account kube-system:kubernetes-dashboard and it says it already exists, i've tried removing it and recreating it and the issue remains.

What you expected to happen:
To be able to browse to our dashboard when calling
az aks browse --resource-group MYRESOURCEGROUP --name my-aks-server
How to reproduce it (as minimally and precisely as possible):
Upgrade a cluster from 1.14.3 -> 1.14.8

Anything else we need to know?:
I was emailed by azure telling me I have to upgrade before the end of this week, but as this has happened on our DEV server, and we require the dashboard for deployments, I can't see myself upgrading the live server.

Environment:

  • 1.14.8
  • 4 Nodes
  • Running 10 x Docker containers processing email jobs from an azure service bus.
  • Code running in linux containers running dotnet core. (Shouldnt be a factor with this issue though)
@inatale
Copy link

inatale commented Feb 18, 2020

I have the same error. I think it related to this issue kubernetes-sigs/metrics-server#188. But I was able to fix metric-server issue.

@markmcgookin
Copy link
Author

I raised a support ticket with MS about this, while it's not a fix, it is a workaround. My issue was all the solutions assumed I was using az commands via powershell locally, but I tend to use the cloud shell, as it's already logged in etc..

For me I had to do the aks browse command, then make the URL like this:

https://gatewayXX.westeurope.console.azure.com/n/xx-XXXXXXXX/xx-XXXXXXXX/proxy/8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/node?namespace=default

Basically just pasting on the api/v1... etc onto the end of the one that command produces.

I then hit an error where I couldn't display nodes once I could see the dashboard, I had to create a cluster role binding, but it didn't work as it already existed. So I had to delete it with

kubectl delete ClusterRoleBinding kubernetes-dashboard

Then recreate it with this

kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

@inatale
Copy link

inatale commented Feb 18, 2020

Yes, thank you a lot. I've found the same workaround and as well "new" one:
Enable the AKS-Preview extension in Cloud Shell: az extension add --name aks-preview (https://github.com/microsoft/MCW-Cloud-native-applications/issues/141).

@squasta
Copy link

squasta commented Apr 22, 2020

I have exactly the same issue with a brand new AKS version 1.16.7. Impossible to get K8S dashboard UI after executing az aks browse .... and connect to http://127.0.0.1:8001

@markmcgookin
Copy link
Author

I will actually close this, as I tried it the other day and it was working fine again - We are on 1.14.8 still though.

@raswinraaj
Copy link

@sq

I have exactly the same issue with a brand new AKS version 1.16.7. Impossible to get K8S dashboard UI after executing az aks browse .... and connect to http://127.0.0.1:8001

@squasta How did you get this resolved? I am facing the same issue after upgrading to 1.16.7 from 1.15.7

@markmcgookin
Copy link
Author

@squasta How did you get this resolved? I am facing the same issue after upgrading to 1.16.7 from 1.15.7

For 1.14.8 it did just "go away" over time, I assume they updated things behind the scenes or did a patch. My comment here provided me with a work around until that happened though.

@ghost ghost locked as resolved and limited conversation to collaborators Jul 22, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

4 participants