Unable to fully scrape metrics from source kubelet_summary

[ajatkj]

Using k8s version 1.11, metric server version 0.3.1.

Deployed metric-server using kubectl create -f deploy/1.8+/

However, on checking metric server pods log, I can see-

I1101 06:08:08.436868 1 round_trippers.go:405] GET https://k8s-node-1:10250/stats/summary/ in 10006 milliseconds
I1101 06:08:08.436919 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.451861 1 round_trippers.go:405] GET https://k8s-node-2:10250/stats/summary/ in 10004 milliseconds
I1101 06:08:08.451896 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.455742 1 round_trippers.go:405] GET https://k8s-master:10250/stats/summary/ in 10003 milliseconds
I1101 06:08:08.455769 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.455816 1 manager.go:150] ScrapeMetrics: time: 10.0274536s, nodes: 0, pods: 0
E1101 06:08:08.455843 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s-node-1: unable to fetch metrics from Kubelet k8s-node-1 (k8s-node-1): Get https://k8s-node-1:10250/stats/summary/: dial tcp: lookup k8s-node-1 on 10.96.0.10:53: read udp 10.244.2.27:54765->10.96.0.10:53: i/o timeout, unable to fully scrape metrics from source kubelet_summary:k8s-node-2: unable to fetch metrics from Kubelet k8s-node-2 (k8s-node-2): Get https://k8s-node-2:10250/stats/summary/: dial tcp: lookup k8s-node-2 on 10.96.0.10:53: read udp 10.244.2.27:40281->10.96.0.10:53: i/o timeout, unable to fully scrape metrics from source kubelet_summary:k8s-master: unable to fetch metrics from Kubelet k8s-master (k8s-master): Get https://k8s-master:10250/stats/summary/: dial tcp: lookup k8s-master on 10.96.0.10:53: read udp 10.244.2.27:56051->10.96.0.10:53: i/o timeout]
I1101 06:08:08.455883 1 manager.go:115] ...Storing metrics...
I1101 06:08:08.455902 1 manager.go:126] ...Cycle complete

I changed metrics-server-deployment.yaml to include below flags:

        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --v=10

When I do curl -k https://k8s-node-2:10250/stats/summary/ I get response as Unauthorized

I am also running heapster for dashboard which shows the stats correctly.

Anybody know what could be the issue? or any other information required.

[ajatkj]

Issue resolved by adding --kubelet-preferred-address-types=InternalIP flag in metrics-server deployment.

[minasys]

@ajatkj I had the same issue. I followed your steps to troubleshoot it and now it is fixed, but when I run "kubectl top node" I get the following error:
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

and when I run "kubectl get hpa", I get the targets unknown like that:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache /50% 1 10 1 1m

Did you have the same issue?

[ankilosaurus]

@minasys check kubectl version. you might need to download newer kubectl and try.

[ankilosaurus]

@ajatkj i am getting same error. What i noticed is metric server is able to get metrics from kubelet of the node which is hosting metric server but not from kubelets of other nodes.

I am already passing those flags but still same issue.

[vikash32]

I am also facing same issue , even if passing those flags still getting below error. i am using 1.12.0v

E0225 17:35:39.347372       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:node2: unable to fetch metrics from Kubelet node2 (10.240.0.22): request failed - "401 Unauthorized", response: "Unauthorized"

[anjuls]

Having same issue on k8s 1.10.0

      - /metrics-server
      - --v=2
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP

Kubelet - removed --read-only-port=0

[GregSilverman]

Sweet! Thanks for the assist @anjuls ... very annoying that this didn't work right when deployed.

[binudba]

K8s v1.16.x and Metrics server 0.3.4, following fix worked.

  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP

[VishnuT-Procore]

I am facing the same issue.
EKS 1.17 and Metrics server 0.3.6 (image : k8s.gcr.io/metrics-server-amd64:v0.3.6)

I have download yaml from [https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml]
and added flag
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

1. kubectl top nodes
   Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

2. kst2 describe apiservice v1beta1.metrics.k8s.io
   Message: failing or missing response from https://:4443/apis/metrics.k8s.io/v1beta1: Get https://:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
   Reason: FailedDiscoveryCheck
   : failing or missing response from https://10.109.108.61:4443/apis/metrics.k8s.io/v1beta1: Get https://10.109.108.61:4443/apis/metrics.k8s.io/v1beta1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
   Reason: FailedDiscoveryCheck

3. kubectl logs -n kube-system deploy/metrics-server
   I0823 18:05:13.762897 1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
   I0823 18:05:14.166947 1 secure_serving.go:116] Serving securely on [::]:4443
   E0823 18:46:15.163853 1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-.ec2.internal: unable to get CPU for container "metrics-server-nanny" in pod kube-system/metrics-server-v0.3.6-859bf9969f-pzt5b on node "", discarding data: missing cpu usage metric

any clue is appreciated. TIA !