Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to fetch pod metrics & request failed - "401 Unauthorized" #212

Open
mabushey opened this issue Jan 29, 2019 · 28 comments

Comments

@mabushey
Copy link

commented Jan 29, 2019

Added "fixes" which reduces the errors:
git diff deploy/1.8+/metrics-server-deployment.yaml

       - name: metrics-server
         image: k8s.gcr.io/metrics-server-amd64:v0.3.1
+        command:
+          - /metrics-server
+          - --kubelet-insecure-tls
+          - --kubelet-preferred-address-types=InternalIP
         imagePullPolicy: Always
         volumeMounts:
         - name: tmp-dir
➜  metrics-server  git:(master) ✗ kubectl top nodes
error: metrics not available yet
➜  metrics-server  git:(master) ✗ kubectl top pod

kubectl -n kube-system logs -f metrics-server-68df9fbc9f-fsvgn

E0129 00:52:04.760832       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-policy-56c4579578-k5szz: no metrics known for pod
E0129 00:52:07.145193       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145211       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/coredns-784bfc9fbd-pw6hz: no metrics known for pod
E0129 00:52:07.145215       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145218       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145221       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-11-127.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145224       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145227       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145230       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145233       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145236       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145239       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-events-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145242       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-9-104.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145244       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-events-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145247       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145250       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145254       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145257       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/dns-controller-7fb44784-np4bd: no metrics known for pod
E0129 00:52:07.145260       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-policy-56c4579578-k5szz: no metrics known for pod
E0129 00:52:07.145263       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-10-132-9-84.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145266       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-events-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145269       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/coredns-784bfc9fbd-q8f52: no metrics known for pod
E0129 00:52:07.145272       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-grafana-post-install-cwx6n: no metrics known for pod
E0129 00:52:07.145277       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-10-132-10-233.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145296       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145305       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-cleanup-secrets-56bjj: no metrics known for pod
E0129 00:52:07.145310       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/tiller-deploy-57f988f854-zjftk: no metrics known for pod
E0129 00:52:07.145318       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-security-post-install-98mvv: no metrics known for pod
E0129 00:52:07.145322       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/metrics-server-68df9fbc9f-fsvgn: no metrics known for pod
E0129 00:52:07.145325       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-10-132-10-63.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:07.145329       1 reststorage.go:144] unable to fetch pod metrics for pod kube-system/etcd-server-ip-10-132-11-28.us-west-2.compute.internal: no metrics known for pod
E0129 00:52:34.895839       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-policy-56c4579578-k5szz: no metrics known for pod
E0129 00:52:48.373899       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-132-9-84.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-9-84.us-west-2.compute.internal (10.132.9.84): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-132-10-63.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-10-63.us-west-2.compute.internal (10.132.10.63): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-132-11-28.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-11-28.us-west-2.compute.internal (10.132.11.28): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-132-9-104.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-9-104.us-west-2.compute.internal (10.132.9.104): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-132-11-127.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-11-127.us-west-2.compute.internal (10.132.11.127): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-132-10-233.us-west-2.compute.internal: unable to fetch metrics from Kubelet ip-10-132-10-233.us-west-2.compute.internal (10.132.10.233): request failed - "401 Unauthorized", response: "Unauthorized"]
E0129 00:53:05.099637       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-policy-56c4579578-k5szz: no metrics known for pod
E0129 00:53:35.216151       1 reststorage.go:144] unable to fetch pod metrics for pod istio-system/istio-policy-56c4579578-k5szz: no metrics known for pod

Is there a version that works (ie one of the 200 forks)?
I've used k8s 1.10 and 1.11 on AWS via Kops.

@zahid0

This comment has been minimized.

Copy link

commented Jan 31, 2019

Metrics server may fail to authenticate if kubelet is running with --anonymous-auth=false flag.
Passing --authentication-token-webhook=true and --authorization-mode=Webhook flags to kubelet can fix this.
kops config for kubelet:

kubelet:
  anonymousAuth: false
  authenticationTokenWebhook: true
  authorizationMode: Webhook

This might break authorization for kubelet-api user if ClusterRoleBinding is not created with system:kubelet-api-admin. Which can be fixed by creating the ClusterRoleBinding:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api-admin
subjects:
- kind: User
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
@githubcdr

This comment has been minimized.

Copy link

commented Feb 3, 2019

@mabushey I believe using "args" is slightly better than "command", it respects the entrypoint.

      - args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
@mabushey

This comment has been minimized.

Copy link
Author

commented Feb 4, 2019

@githubcdr
Thanks for the comment, I agree that this seems better, however command works, args does not.

@rajeshkodali

This comment has been minimized.

Copy link

commented Feb 9, 2019

I have created metric server with below deployment and addedd kubelet config in kops but I still get 401

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.3.1
        imagePullPolicy: Always
        command:
            - /metrics-server
            - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
            - --kubelet-insecure-tls
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

Logs:

E0209 22:52:55.288570       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-64-197.compute.internal: unable to fetch metrics from Kubelet ip-172-20-64-197.compute.internal (172.20.64.197): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-20-100-28.compute.internal: unable to get CPU for container "sentinel" in pod default/redis-sentinel-744bj on node "172.20.100.28", discarding data: missing cpu usage metric]
E0209 22:53:55.273084       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-117-178.us-west-2.compute.internal: unable to get CPU for container "nginx-ingress" in pod default/nginx-ingress-rc-gj9tl on node "172.20.117.178", discarding data: missing cpu usage metric, unable to fully scrape metrics from source kubelet_summary:ip-172-20-64-197.compute.internal: unable to fetch metrics from Kubelet ip-172-20-64-197.compute.internal (172.20.64.197): request failed - "401 Unauthorized", response: "Unauthorized"]
E0209 22:54:55.286313       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-172-20-64-197.compute.internal: unable to fetch metrics from Kubelet ip-172-20-64-197.compute.internal (172.20.64.197): request failed - "401 Unauthorized", response: "Unauthorized"
E0209 22:56:55.264838       1 manager.go:102] unable to fully collect metrics: unable to extract connection information for node "ip-172-20-69-45.compute.internal": node ip-172-20-69-45.compute.internal is not ready
E0209 22:57:55.266908       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-172-20-69-45.compute.internal: unable to get CPU for container "kafka" in pod default/kafka-1 on node "172.20.69.45", discarding data: missing cpu usage metric
E0209 22:59:55.255294       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-172-20-117-178.compute.internal: unable to get CPU for container "nginx-ingress" in pod default/nginx-ingress-rc-gj9tl on node "172.20.117.178", discarding data: missing cpu usage metric
E0209 23:03:52.091447       1 reststorage.go:144] unable to fetch pod metrics for pod default/baker-xxx: no metrics known for pod
E0209 23:05:55.297098       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-172-20-117-178.compute.internal: unable to get CPU for container "nginx-ingress" in pod default/nginx-ingress-rc-gj9tl on node "172.20.117.178", discarding data: missing cpu usage metric
@serathius

This comment has been minimized.

Copy link
Contributor

commented Feb 11, 2019

@rajeshkodali

This comment has been minimized.

Copy link

commented Feb 11, 2019

It works now. I am not sure what caused it to work.

kubectl describe clusterrole system:metrics-server
Name:         system:metrics-server
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"system:metrics-server","namespace":""},"rules":[...
PolicyRule:
  Resources               Non-Resource URLs  Resource Names  Verbs
  ---------               -----------------  --------------  -----
  namespaces              []                 []              [get list watch]
  nodes/stats             []                 []              [get list watch]
  nodes                   []                 []              [get list watch]
  pods                    []                 []              [get list watch]
  deployments.extensions  []                 []              [get list watch] 

mattkelly added a commit to mattkelly/metrics-server that referenced this issue Feb 12, 2019

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 12, 2019

@zahid0 I'm still facing the issue, I tried to edit cluster and update the metric yaml file. Below are my changes:

# kops edit cluster
  kubeAPIServer:
    kubeletPreferredAddressTypes:
    - InternalIP
    - Hostname
    - InternalDNS
    - ExternalDNS
    - ExternalIP
    runtimeConfig:
      autoscaling/v2beta1: "true"
  kubeControllerManager:
    horizontalPodAutoscalerUseRestClients: false
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
# metrics-server.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        command:
          - /metrics-server
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubelet-api-admin
subjects:
- kind: User
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io

@rajeshkodali I also ran your command, I don't think it works

kubectl describe clusterrole system:metrics-server

Name:         system:metrics-server
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources    Non-Resource URLs  Resource Names  Verbs
  ---------    -----------------  --------------  -----
  nodes/stats  []                 []              [get list watch]
  nodes        []                 []              [get list watch]
  pods         []                 []              [get list watch]

@slashr

This comment has been minimized.

Copy link

commented Feb 12, 2019

I'm facing the same problems as @vinhnglx despite making all the fixes mentioned on this issue. :(

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 13, 2019

guys, any idea? I spent a few hours today but still can't make it works.

@rajeshkodali

This comment has been minimized.

Copy link

commented Feb 13, 2019

Here is my working config:
kops cluster spec for kubelet.

  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook

Metrics server yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - nodes
      - nodes/stats
      - namespaces
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - deployments
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.3.1
        imagePullPolicy: Always
        command:
            - /metrics-server
            - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
            - --kubelet-insecure-tls
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

role.yaml from kubernetes/kops#5706 and

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:kubelet-api-admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
subjects:
- kind: User
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io
@vinhnglx

This comment has been minimized.

Copy link

commented Feb 14, 2019

thanks @rajeshkodali .

I still hit the error "401 Unauthorized"

E0214 03:16:54.413600       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-10-2-189.ap-southeast-1.compute.internal: unable to fetch metrics from Kubelet ip-10-10-2-189.ap-southeast-1.compute.internal (10.10.2.189): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-10-1-140.ap-southeast-1.compute.internal: unable to fetch metrics from Kubelet ip-10-10-1-140.ap-southeast-1.compute.internal (10.10.1.140): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-10-10-1-124.ap-southeast-1.compute.internal: unable to fetch metrics from Kubelet ip-10-10-1-124.ap-southeast-1.compute.internal (10.10.1.124): request failed - "401 Unauthorized", response: "Unauthorized"]
E0214 03:17:03.349620       1 reststorage.go:144] unable to fetch pod metrics for pod default/backend-14-feb-2019-10-20-15-5bb5b77bcc-stb4t: no metrics known for pod
E0214 03:17:17.633193       1 reststorage.go:144] unable to fetch pod metrics for pod default/frontend-14-feb-2019-10-20-04-5d5c4678bc-k7vpv: no metrics known for pod
E0214 03:17:33.357307       1 reststorage.go:144] unable to fetch pod metrics for pod default/backend-14-feb-2019-10-20-15-5bb5b77bcc-stb4t: no metrics known for pod
@rajeshkodali

This comment has been minimized.

Copy link

commented Feb 14, 2019

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 14, 2019

The output when running that command: error: metrics not available yet

0214 11:32:45.819869   83755 loader.go:359] Config loaded from file /Users/developers/.kube/config
I0214 11:32:45.822017   83755 loader.go:359] Config loaded from file /Users/developers/.kube/config
I0214 11:32:45.822606   83755 round_trippers.go:419] curl -k -v -XGET  -H "User-Agent: kubectl/v1.13.2 (darwin/amd64) kubernetes/cff46ab" -H "Accept: applicat
ion/json, */*" -H "Authorization: Basic xxxxxxxxxx=" 'https://api.xxx.xxx.com/api?timeout=32s
'
I0214 11:32:45.870240   83755 round_trippers.go:438] GET https://api.xxx.xxx.com/api?timeout=32s 200 OK in 47 milliseconds
I0214 11:32:45.870286   83755 round_trippers.go:444] Response Headers:
I0214 11:32:45.870299   83755 round_trippers.go:447]     Content-Type: application/json
I0214 11:32:45.870313   83755 round_trippers.go:447]     Content-Length: 133
I0214 11:32:45.870323   83755 round_trippers.go:447]     Date: Thu, 14 Feb 2019 03:32:45 GMT
I0214 11:32:45.870412   83755 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","
serverAddress":"10.10.1.124:443"}]}
I0214 11:32:45.870842   83755 round_trippers.go:419] curl -k -v -XGET  -H "Authorization: Basic xxxxxx=" -H "Acce
pt: application/json, */*" -H "User-Agent: kubectl/v1.13.2 (darwin/amd64) kubernetes/cff46ab" 'https://api.xxx.xxx.com/apis?timeout=32
s'
I0214 11:32:45.880593   83755 round_trippers.go:438] GET https://api.xxx.xxx.com/apis?timeout=32s 200 OK in 9 milliseconds
I0214 11:32:45.880617   83755 round_trippers.go:444] Response Headers:
I0214 11:32:45.880627   83755 round_trippers.go:447]     Content-Type: application/json
I0214 11:32:45.880636   83755 round_trippers.go:447]     Content-Length: 3609
I0214 11:32:45.880645   83755 round_trippers.go:447]     Date: Thu, 14 Feb 2019 03:32:45 GMT
I0214 11:32:45.880709   83755 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"},{"groupVersion":"authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"},{"groupVersion":"autoscaling/v2beta1","version":"v2beta1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"}},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"}},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"}},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"}},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"},{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"}},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"}},{"name":"admissionregistration.k8s.io","versions":[{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}},{"name":"scheduling.k8s.io","versions":[{"groupVersion":"scheduling.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"scheduling.k8s.io/v1beta1","version":"v1beta1"}},{"name":"metrics.k8s.io","versions":[{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}}]}
I0214 11:32:45.896066   83755 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.2 (darwin/amd64) kubernetes/cff46ab" -H "Authorization: Basic xxxxxxx=" 'https://api.xxx.xxx.com/apis/metrics.k8s.io/v1beta1/nodes'
I0214 11:32:45.906012   83755 round_trippers.go:438] GET https://api.xxx.xxx.com/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 9 milliseconds
I0214 11:32:45.906056   83755 round_trippers.go:444] Response Headers:
I0214 11:32:45.906107   83755 round_trippers.go:447]     Date: Thu, 14 Feb 2019 03:32:45 GMT
I0214 11:32:45.906134   83755 round_trippers.go:447]     Content-Length: 137
I0214 11:32:45.906165   83755 round_trippers.go:447]     Content-Type: application/json
I0214 11:32:45.906205   83755 request.go:942] Response Body: {"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[]}
F0214 11:32:45.907600   83755 helpers.go:116] error: metrics not available yet
@zahid0

This comment has been minimized.

Copy link

commented Feb 21, 2019

@vinhnglx could you check the arguments passed to kubelet on one of the nodes? If you run kubelet using systemd, then ssh to instance and run sudo systemctl status kubelet. Make sure --authentication-token-webhook=true and --authorization-mode=Webhook flags are passed.
Checking kubectl logs may also help (run journalctl -u kubelet on the node for logs).

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 22, 2019

@zahid0 I'm using Kops to install Kubernetes with vpc, private subnets, and calico CNI for networking. I'm not able to ssh to the instance to check the kubelet.

But I already set the authentication-token-webhook=true and authorization-mode=Webhook using kops edit cluster command

# kops edit cluster

kind: Cluster
metadata:
  creationTimestamp: 2019-01-29T06:45:14Z
  name: xxx.xxx.com
spec:
  # ...
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook  

And it still shows the 401 Unauthorized

@zahid0

This comment has been minimized.

Copy link

commented Feb 22, 2019

@vinhnglx do you mind showing the output of kops update cluster and kops rolling-update cluster ...

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 22, 2019

@zahid0 I don't mind, but I'm using kops with terraform output.

My steps are:

  • edit cluster
  • update cluster output to Terraform files
  • terraform plan and terraform apply
@zahid0

This comment has been minimized.

Copy link

commented Feb 22, 2019

@vinhnglx kops rolling-update is required after terraform apply. https://github.com/kubernetes/kops/blob/master/docs/terraform.md#caveats

@vinhnglx

This comment has been minimized.

Copy link

commented Feb 22, 2019

@zahid0 Oh, no. My script has a mistake, should run kops rolling-update but I ran kops update. Now it works.

I can get the metrics. Thanks a lot for your help :)

kubectl top nodes
NAME                                             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-1-2-3-4.ap-southeast-1.compute.internal   160m         8%     1585Mi          41%
ip-1-2-3-4.ap-southeast-1.compute.internal   1151m        57%    2397Mi          30%
ip-1-2-3-4.ap-southeast-1.compute.internal   1005m        50%    2769Mi          35%
@khteh

This comment has been minimized.

Copy link

commented Mar 12, 2019

I changed from using "args" to "commands" and I don't see the 401 Unauthorized now. However, kubectl logs -f metrics-server... -n kube-system still shows "no metrics known for pod":

$ k logs -f metrics-server-68df9fbc9f-dgr8v -n kube-system
I0312 03:55:41.841800       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0312 03:55:42.433339       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0312 03:55:42.439873       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0312 03:55:42.488139       1 serve.go:96] Serving securely on [::]:443
E0312 03:55:46.554516       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-0: no metrics known for pod
E0312 03:55:46.554540       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-1: no metrics known for pod
  E0312 05:08:42.634201       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-192-168-84-18.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-connector" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:ip-192-168-22-244.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-converse" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-admin" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-ui" in pod default/iconverse-ui-0 on node "192.168.22.244", discarding data: missing cpu usage metric]]

kubectl top nodes show valid data with resource percentage. kubectl top pod does not have any percentage at all.

@khteh

This comment has been minimized.

Copy link

commented Mar 12, 2019

@zahid0, how to add the kubelet config when I am using eksctl to create the cluster on AWS EKS?

@mingfang

This comment has been minimized.

@zoltan-fedor

This comment has been minimized.

Copy link

commented Jun 10, 2019

I experience the same as @khteh above.

I have set the following kubelet arguments:

--authentication-token-webhook
--authorization-mode Webhook

The kubectl top nodes show valid data with resource percentage. kubectl top pods does not have any percentage, but shows values.

The metrics-server logs:

$ k logs -n kube-system metrics-server-5dddc64cb8-t5bqp
I0610 20:40:55.524499       1 serving.go:273] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
W0610 20:40:56.190762       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0610 20:40:56.199965       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/06/10 20:40:56 log.go:33: [restful/swagger] listing is available at https://:8443/swaggerapi
[restful] 2019/06/10 20:40:56 log.go:33: [restful/swagger] https://:8443/swaggerui/ is mapped to folder /swagger-ui/
I0610 20:40:56.251231       1 serve.go:96] Serving securely on [::]:8443
E0610 20:41:28.930599       1 reststorage.go:148] unable to fetch pod metrics for pod default/myref-res-search-55bb66d4c4-6689m: no metrics known for pod
E0610 20:41:43.942248       1 reststorage.go:148] unable to fetch pod metrics for pod default/myref-res-search-55bb66d4c4-6689m: no metrics known for pod

I am using AWS EKS with Kubernetes 1.12 and installed the metric-server from helm (https://github.com/helm/charts/tree/master/stable/metrics-server):

$ helm install stable/metrics-server \
    --name metrics-server \
    --version 2.8.2 \
    --namespace kube-system \
    --set args={"--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP"}

The pod is deployed with a HorizontalPodAutoscaler:

{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "myref-resolution-search.fullname" . }}
  labels:
    app: {{ include "myref-resolution-search.fullname" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    heritage: {{ .Release.Service }}
    release: {{ .Release.Name }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "myref-resolution-search.fullname" . }}
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}

Still no percentages:

$ k top pods
NAME                                       CPU(cores)   MEMORY(bytes)   
myref-res-search-55bb66d4c4-49qpx   137m         1027Mi          

Any idea why the kubectl top pods shows no percentage at all?

@zhanghan12

This comment has been minimized.

Copy link

commented Jun 25, 2019

E0625 13:13:04.145733       1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:172.20.0.217: unable to get CPU for container "java-springboot-web" in pod java-springboot-web-b8cff79f5-w9bqq on node "172.20.0.217", discarding data: missing cpu usage metric

The kubectl top nodes show valid data with resource percentage. kubectl top pods does not have any percentage, but shows values.

[root@blv0155 metrics]# kubectl  top pod -n smix3 
NAME                                        CPU(cores)   MEMORY(bytes)   
java-springboot-web-b8cff79f5-vknhh   3m           428Mi        
[root@blv0155 metrics]# kubectl  top node
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
172.17.71.208   51m          2%     1933Mi          52%       
172.17.71.209   114m         2%     1862Mi          24%     
https://172.17.71.208:6443/apis/metrics.k8s.io/v1beta1/pods
      "metadata": {
        "name": "java-springboot-web-b8cff79f5-vknhh",
        "namespace": "smix3",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/smix3/pods/java-springboot-web-b8cff79f5-vknhh",
        "creationTimestamp": "2019-06-25T13:38:35Z"
      },
      "timestamp": "2019-06-25T13:38:32Z",
      "window": "30s",
      "containers": [
        {
          "name": "tutor-java-springboot-web",
          "usage": {
            "cpu": "982317n",
            "memory": "470060Ki"
          }
        }
      ]
    },
@k4korea

This comment has been minimized.

Copy link

commented Jun 26, 2019

Deployment , Replicaset yaml

### spec:
Resource :
request :
cpu :

yaml write have to
metric info unknown -> 20%, 30% change

@sreedharbukya

This comment has been minimized.

Copy link

commented Jul 12, 2019

Here is my working config:
kops cluster spec for kubelet.

  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook

Metrics server yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - nodes
      - nodes/stats
      - namespaces
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - deployments
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.3.1
        imagePullPolicy: Always
        command:
            - /metrics-server
            - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
            - --kubelet-insecure-tls
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

role.yaml from kubernetes/kops#5706 and

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:kubelet-api-admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
subjects:
- kind: User
  name: kubelet-api
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:kubelet-api-admin
  apiGroup: rbac.authorization.k8s.io

This configuration worked me too. It took a lot of time to completely rollout for my cluster.

@sreedharbukya

This comment has been minimized.

Copy link

commented Jul 22, 2019

I got it working. Thanks, Team.

@tcolgate

This comment has been minimized.

Copy link

commented Jul 26, 2019

should we aggregate all of these setup problems into one ticket? e.g. #278
Also, it might be an idea to have an option to make node stats requests via the api proxy. That would avoid the need for users to change kubelet auth setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.