Compatibility with Kubernetes 1.10

[richerlariviere]

Environment

Dashboard version: v1.8.3
Kubernetes version: v1.10.1
Operating system: macOS

Steps to reproduce

This is more a question rather than an issue. I configured a brand new cluster with Kubernetes 1.10.1 and I wanted to know if this version is fully supported at this moment. If I look at the compatibility matrix from the wiki, I can't see any reference to K8s 1.10.

If this K8s 1.10 is supported (in that case we'll have to change the issue name also), then follow those steps:

1. Create a basic cluster. I assume the user you use has a ClusterRoleBinding assigned to cluster-admin. This user uses an openID authentication using Azure. I'm using RBAC with aggregated apis which means I set requestheader-allowed-names, requestheader-client-ca-file, requestheader-extra-headers-prefix, requestheader-group-headers, requestheader-username-headers flags.

2. Install the dashboard (kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml)

3. Access the dashboard using kubectl proxy. I don't have the login/skip part of the dashboard configuration as I can already access everything.

4. Go to setting page (http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/settings?namespace=kube-system)

Observed result

An error is displayed:

Expected result

I should have access to the settings page as my user is a cluster-admin and cluster-admin is in the system:masters Group. I guess it's a problem with Dashboard RBAC because I can modify the configmap containing the same parameters as the setting page using:

$ kubectl edit configmaps/kubernetes-dashboard-settings -n kube-system

Comments

Dashboard ClusterRoleBinding

Name:         kubernetes-dashboard
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              k8s-app=kubernetes-dashboard
              kubernetes.io/cluster-service=true
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":...
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name                  Namespace
  ----            ----                  ---------
  ServiceAccount  kubernetes-dashboard  kube-system

Dashboard Service

Name:                     kubernetes-dashboard
Namespace:                kube-system
Labels:                   k8s-app=kubernetes-dashboard
                          kubernetes.io/cluster-service=true
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"k8s-app":"kubernetes-dashboard","kubernetes.io/cluster-service":"true"},"na...
Selector:                 k8s-app=kubernetes-dashboard
Type:                     NodePort
IP:                       10.0.70.8
Port:                     <unset>  80/TCP
TargetPort:               9090/TCP
NodePort:                 <unset>  31639/TCP
Endpoints:                10.244.0.8:9090
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Dashboard Deployment

Name:                   kubernetes-dashboard
Namespace:              kube-system
CreationTimestamp:      Thu, 19 Apr 2018 14:00:32 -0400
Labels:                 addonmanager.kubernetes.io/mode=Reconcile
                        k8s-app=kubernetes-dashboard
                        kubernetes.io/cluster-service=true
Annotations:            deployment.kubernetes.io/revision=3
                        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"...
Selector:               k8s-app=kubernetes-dashboard
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           k8s-app=kubernetes-dashboard
  Service Account:  kubernetes-dashboard
  Containers:
   kubernetes-dashboard:
    Image:      k8s-gcrio.azureedge.net/kubernetes-dashboard-amd64:v1.8.3
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --heapster-host=http://heapster.kube-system:80
    Limits:
      cpu:     300m
      memory:  150Mi
    Requests:
      cpu:        300m
      memory:     150Mi
    Liveness:     http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-dashboard-64dcf5784f (1/1 replicas created)
Events:          <none>

[kachkaev]

@richerlariviere have you tied

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

instead of

http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

?

https://github.com/kubernetes/dashboard#getting-started

[richerlariviere]

I'm getting this error:

Error: 'tls: oversized record received with length 20527'
Trying to reach: 'https://10.244.0.8:9090/'

From my research it seems to be a pretty common issue but I found no real solution. Using http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/ is good enough for my needs since I only access the dashboard through kube proxy.

[xmik]

I've set up dashboard on a brand new cluster with Kubernetes 1.10.1, without any authentication or security whatsoever (for now) and it worked.

However, there are no metrics shown from Heapster. I am confused, whether Heapster is deprecated for kubernetes >= 1.8? It is, according to: kubernetes-retired/heapster#1840 (comment) . Will the dashboard support metrics-server instead of Heapster now?

[ErickWendel]

I try to connect on http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/namespace?namespace=default and work for me.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

kubectl proxy

[discostur]

@richerlariviere getting the same error as you. Dashboard Login with Token (Admin Service Account) worked without any problem with k8s 1.9.x and since 1.10.x i'm getting the "you do not have permission" error.

[discostur]

@floreks is there any known login/permission bug with dashboard and k8s 1.10.x?

[suneeta-mall]

I am seeing this problem too (on 1.10.2) .. looks like dashboard cant authorize correctly with cluster and all API responses fails

2018/05/15 07:17:33 Starting overwatch
2018/05/15 07:17:33 Using in-cluster config to connect to apiserver
2018/05/15 07:17:33 Using service account token for csrf signing
2018/05/15 07:17:33 No request provided. Skipping authorization
2018/05/15 07:17:33 Successful initial request to the apiserver, version: v1.10.2
2018/05/15 07:17:33 Generating JWE encryption key
2018/05/15 07:17:33 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/05/15 07:17:33 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/05/15 07:17:33 Initializing JWE encryption key from synchronized object
2018/05/15 07:17:33 Creating in-cluster Heapster client
2018/05/15 07:17:33 Auto-generating certificates
2018/05/15 07:17:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:17:33 Successfully created certificates
2018/05/15 07:17:33 Serving securely on HTTPS port: 8443
2018/05/15 07:18:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:18:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:19:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:19:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:20:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:20:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:21:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:21:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:22:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:22:33 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:23:03 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2018/05/15 07:23:22 Getting application global configuration
2018/05/15 07:23:22 Application configuration {"serverTime":1526369002907}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Outcoming response to 172.20.37.14:10818 with 200 status code
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/systembanner request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Outcoming response to 172.20.37.14:10818 with 200 status code
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Outcoming response to 172.20.37.14:10818 with 200 status code
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Outcoming response to 172.20.37.14:10818 with 200 status code
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Outcoming response to 172.20.37.14:10818 with 200 status code
2018/05/15 07:23:23 [2018-05-15T07:23:23Z] Incoming HTTP/2.0 GET /api/v1/overview?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 172.20.37.14:10818: {}
2018/05/15 07:23:23 Getting config category
2018/05/15 07:23:23 Non-critical error occurred during resource retrieval: configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list configmaps at the cluster scope

[xmik]

@suneeta-mall your problem seems like a different one. Your log messages contain:

Successful initial request to the apiserver, version: v1.10.2

which says connection to apiserver is fine.

Do you have Heapster running? Maybe this comment will help you?

I managed to get kube-dashboard working with secure apiserver 1.10.2 and Heapster gcr.io/google_containers/heapster:v1.5.2. I can see the metrics on kube-dashboard. (Edit: but I didn't have the Forbidden (403) error, only problem with kube-dashboard showing no metrics from running and reachable Heapster)

[suneeta-mall]

@xmik Yes I setup heapster after the fact but thats not related to forbidden log at the end of log message .. My understanding of the situation is that connection to API server is fine resource access ex forbidden

[xmik]

@suneeta-mall Can you try if applying this (with kubectl apply -f) resolves the problem?

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-list-configmaps
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dashboard-list-configmaps-bind-to-dashboard-sa
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard-list-configmaps
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

[suneeta-mall]

@xmik Sounds about right ... Using ClusterRole instead of Role (as used in recommended dashboard spec) seems to do the trick. Now I am confused why recommended dashboard RBAC is Role* and not ClusterRole* ... My reading of this is to control access by namespace (same or all) .. and dashboard should have all ? Perhaps I need to read more on the difference between these two.

Thanks for the tip :)

EDIT: I can confirm though using ClusterRole with explicit verb (ex list) as suggested by @xmik fixes the dashboard but I am not sure why thats a problem with 1.10 .. I have another kube cluster on 1.9.x with RBAC Role on configmap without list and it seems to work fine.

[richerlariviere]

I tested it on Kubernetes 1.10.3 and I got the same problem (even though my Dashboard is binded with cluster-admin clusterrolebinding).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[richerlariviere]

I'm closing this issue as for now the discussion provides a temporary workaround before we get an update from the Kubernetes Dashboard project.

Please follow issue #2986 if you want to track the switch from Heapster to Metrics API.