Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to fully scrape metrics from source kubelet_summary #167

Closed
ajatkj opened this issue Nov 1, 2018 · 8 comments
Closed

Unable to fully scrape metrics from source kubelet_summary #167

ajatkj opened this issue Nov 1, 2018 · 8 comments

Comments

@ajatkj
Copy link

@ajatkj ajatkj commented Nov 1, 2018

Using k8s version 1.11, metric server version 0.3.1.

Deployed metric-server using kubectl create -f deploy/1.8+/

However, on checking metric server pods log, I can see-

I1101 06:08:08.436868 1 round_trippers.go:405] GET https://k8s-node-1:10250/stats/summary/ in 10006 milliseconds
I1101 06:08:08.436919 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.451861 1 round_trippers.go:405] GET https://k8s-node-2:10250/stats/summary/ in 10004 milliseconds
I1101 06:08:08.451896 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.455742 1 round_trippers.go:405] GET https://k8s-master:10250/stats/summary/ in 10003 milliseconds
I1101 06:08:08.455769 1 round_trippers.go:411] Response Headers:
I1101 06:08:08.455816 1 manager.go:150] ScrapeMetrics: time: 10.0274536s, nodes: 0, pods: 0
E1101 06:08:08.455843 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:k8s-node-1: unable to fetch metrics from Kubelet k8s-node-1 (k8s-node-1): Get https://k8s-node-1:10250/stats/summary/: dial tcp: lookup k8s-node-1 on 10.96.0.10:53: read udp 10.244.2.27:54765->10.96.0.10:53: i/o timeout, unable to fully scrape metrics from source kubelet_summary:k8s-node-2: unable to fetch metrics from Kubelet k8s-node-2 (k8s-node-2): Get https://k8s-node-2:10250/stats/summary/: dial tcp: lookup k8s-node-2 on 10.96.0.10:53: read udp 10.244.2.27:40281->10.96.0.10:53: i/o timeout, unable to fully scrape metrics from source kubelet_summary:k8s-master: unable to fetch metrics from Kubelet k8s-master (k8s-master): Get https://k8s-master:10250/stats/summary/: dial tcp: lookup k8s-master on 10.96.0.10:53: read udp 10.244.2.27:56051->10.96.0.10:53: i/o timeout]
I1101 06:08:08.455883 1 manager.go:115] ...Storing metrics...
I1101 06:08:08.455902 1 manager.go:126] ...Cycle complete

I changed metrics-server-deployment.yaml to include below flags:

        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --v=10

When I do curl -k https://k8s-node-2:10250/stats/summary/ I get response as Unauthorized

I am also running heapster for dashboard which shows the stats correctly.

Anybody know what could be the issue? or any other information required.

@ajatkj

This comment has been minimized.

Copy link
Author

@ajatkj ajatkj commented Nov 1, 2018

Issue resolved by adding --kubelet-preferred-address-types=InternalIP flag in metrics-server deployment.

@ajatkj ajatkj closed this Nov 1, 2018
@minasys

This comment has been minimized.

Copy link

@minasys minasys commented Dec 4, 2018

@ajatkj I had the same issue. I followed your steps to troubleshoot it and now it is fixed, but when I run "kubectl top node" I get the following error:
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

and when I run "kubectl get hpa", I get the targets unknown like that:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache /50% 1 10 1 1m

Did you have the same issue?

@aksgithub

This comment has been minimized.

Copy link

@aksgithub aksgithub commented Feb 16, 2019

@minasys check kubectl version. you might need to download newer kubectl and try.

@aksgithub

This comment has been minimized.

Copy link

@aksgithub aksgithub commented Feb 16, 2019

@ajatkj i am getting same error. What i noticed is metric server is able to get metrics from kubelet of the node which is hosting metric server but not from kubelets of other nodes.

I am already passing those flags but still same issue.

@vikash32

This comment has been minimized.

Copy link

@vikash32 vikash32 commented Feb 25, 2019

I am also facing same issue , even if passing those flags still getting below error. i am using 1.12.0v

E0225 17:35:39.347372       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:node2: unable to fetch metrics from Kubelet node2 (10.240.0.22): request failed - "401 Unauthorized", response: "Unauthorized"
@anjuls

This comment has been minimized.

Copy link

@anjuls anjuls commented Mar 6, 2019

Having same issue on k8s 1.10.0

      - /metrics-server
      - --v=2
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP

Kubelet - removed --read-only-port=0

@GregSilverman

This comment has been minimized.

Copy link

@GregSilverman GregSilverman commented Apr 30, 2019

Sweet! Thanks for the assist @anjuls ... very annoying that this didn't work right when deployed.

gazal-k added a commit to gazal-k/eks-quickstart-app-dev that referenced this issue Sep 27, 2019
issue reference: kubernetes-sigs/metrics-server#167
also bump chart version to 2.8.6
@binudba

This comment has been minimized.

Copy link

@binudba binudba commented Nov 24, 2019

K8s v1.16.x and Metrics server 0.3.4, following fix worked.

  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.