New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HorizontalPodAutoscaler FailedGetResourceMetric, "unable to get metrics for resource cpu" #3006

Closed
rafi opened this Issue Jul 17, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@rafi
Copy link

rafi commented Jul 17, 2018

Horizontal Pod Autoscaler (HPA) fails to collect metrics after setup, for example:

$ kubectl get --all-namespaces event
NAMESPACE      LAST SEEN   FIRST SEEN   COUNT     NAME                                    KIND                      SUBOBJECT   TYPE      REASON                         SOURCE                      MESSAGE
istio-system   1m          11d          34210     istio-egressgateway.153e74efecb3de28    HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   6m          11d          34200     istio-ingress.153e74eff8ffd9e9          HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   1m          11d          34210     istio-ingress.153e74eff95336d2          HorizontalPodAutoscaler               Warning   FailedComputeMetricsReplicas   horizontal-pod-autoscaler   failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   1m          11d          34210     istio-ingressgateway.153e74f0059bb205   HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

BUG REPORT

Environment:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:59:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: VMWare

  • OS:

Linux 3.10.0-862.el7.x86_64 x86_64
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Applications"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
  • Version of Ansible (ansible --version): 2.5.5

Kubespray version (commit): 0b939a49

Network plugin used: Calico

Copy of your inventory file:

[all]
vrnv02ax04315 	 ansible_host=10.213.236.115 ip=10.213.236.115
vrnv02ax04316 	 ansible_host=10.213.236.116 ip=10.213.236.116
vrnv02ax04317 	 ansible_host=10.213.236.117 ip=10.213.236.117
vrnv02ax04318 	 ansible_host=10.213.236.118 ip=10.213.236.118

[kube-master]
vrnv02ax04315 	 
vrnv02ax04316 	 

[kube-node]
vrnv02ax04315 	 
vrnv02ax04316 	 
vrnv02ax04317 	 
vrnv02ax04318 	 

[etcd]
vrnv02ax04315 	 
vrnv02ax04316 	 
vrnv02ax04317 	 

[k8s-cluster:children]
kube-node 	 
kube-master 	 

[calico-rr]

[vault]
vrnv02ax04315 	 
vrnv02ax04316 	 
vrnv02ax04317 	 

Command used to invoke ansible:

ansible-playbook -i inventory/stg/hosts.ini cluster.yml -b -u kube --private-key=~/.ssh/kube_rsa -v
@monkeydaichan

This comment has been minimized.

Copy link

monkeydaichan commented Jul 31, 2018

You deployed "metrics-server"?
cloudfoundry-incubator/kubo-deployment#263

$ git clone https://github.com/kubernetes-incubator/metrics-server.git
$ cd metrics-server
$ kubectl create -f deploy/1.8+/

@rafi

This comment has been minimized.

Copy link

rafi commented Aug 9, 2018

@monkeydaichan yes.

It started working when I upgraded to Istio 1.0.0

@rafi rafi closed this Aug 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment