Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.3.0/0.3.1 unable to fetch metrics from Kubelet: 401 Unauthorized #144

Closed
gjmzj opened this issue Sep 30, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@gjmzj
Copy link

commented Sep 30, 2018

  • k8s version: 1.11.3
  • metrics-server: 0.3.1

error log of 'metrics-server' POD:

E0930 02:45:31.619297       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:10.xx.yy.41: unable to fetch metrics from Kubelet 10.xx.yy.41 (10.xx.yy.41): request failed - "401 Unauthorized", response: "Unauthorized"

kubelet is running with flags '--anonymous-auth=false'

my question is 'how can the metrics-server get authorized by the kubelet server '

did i miss something ?

@gjmzj

This comment has been minimized.

Copy link
Author

commented Oct 1, 2018

through manifests in deploy/1.8+
and also i did a little changes in metrics-server-deployment.yaml :

      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      - name: ssl-dir
        secret:
          secretName: metrics-server-secrets
          defaultMode: 0400
      containers:
      - name: metrics-server
        #image: k8s.gcr.io/metrics-server-amd64:v0.3.0
        image: mirrorgooglecontainers/metrics-server-amd64:v0.3.1
        imagePullPolicy: IfNotPresent
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        - --logtostderr=true
        - --tls-cert-file=/etc/ssl/ms-cert
        - --tls-private-key-file=/etc/ssl/ms-key
        - --v=2
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
        - name: ssl-dir
          mountPath: /etc/ssl

i created 'metrics-server-secrets' before deployment

kubectl create secret generic -n kube-system metrics-server-secrets \
            --from-file=ca=ca.pem \
            --from-file=ms-key=metrics-server-key.pem \
            --from-file=ms-cert=metrics-server.pem"
@DirectXMan12

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2018

metrics-server attempts to authorize itself using token authentication. Please ensure that you're running your kubelets with webhook token authentication turned on.

@gjmzj

This comment has been minimized.

Copy link
Author

commented Oct 4, 2018

thanks, i'll check on it.

@erhudy

This comment has been minimized.

Copy link

commented Oct 9, 2018

I'm experiencing the same issue, but webhook authentication looks to be on to me:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
  webhook:
    enabled: true
  anonymous:
    enabled: false
authorization:
  mode: Webhook
clusterDNS:
  - 10.96.0.10
clusterDomain: cluster.local
kubeReserved:
  cpu: 100m
  memory: 256Mi
rotateCertificates: true
staticPodPath: /etc/kubernetes/manifests
@gjmzj

This comment has been minimized.

Copy link
Author

commented Oct 26, 2018

it works on my cluster when i add following settings on kubelet:

  --authentication-token-webhook \
  --authorization-mode=Webhook \

thanks @DirectXMan12

@geoxanadu

This comment has been minimized.

Copy link

commented Oct 26, 2018

Sorry for the stupid question, how can I access/edit the kubelet? @gjmzj

@DirectXMan12

This comment has been minimized.

Copy link
Contributor

commented Nov 8, 2018

you'll need to figure out how your kubelet is being run, unfortunately I can't help much with that without more information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.