New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics server api not getting registered #45

Closed
sujithvs74 opened this Issue Mar 22, 2018 · 11 comments

Comments

Projects
None yet
10 participants
@sujithvs74

sujithvs74 commented Mar 22, 2018

I have deployed metrics api in kubernetes following https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy/1.8%2B. Metric server is running fine and but I am not able to get metrics from it. I am using kubernetes 1.9 version.

[demo@dev-demo metrics-server]$ kubectl get --raw "/apis/metrics.k8s.io" Error from server (NotFound): the server could not find the requested resource [demo@dev-demo metrics-server]$

`[demo@dev-demo metrics-server]$ k get hpa -n default
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache / 50% 1 10 1 19h
[demo@dev-demo metrics-server]$ k describe hpa -n default
Name: php-apache
Namespace: default
Labels:
Annotations:
CreationTimestamp: Wed, 21 Mar 2018 04:57:32 -0400
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message


AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message


Warning FailedComputeMetricsReplicas 43m (x13 over 49m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 4m (x91 over 49m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)`

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Mar 23, 2018

if you do kubectl get apiservice v1beta1.metrics.k8s.io -o yaml, does the status look ok?

@groob

This comment has been minimized.

groob commented Apr 10, 2018

Same issue as @sujithvs74

status:
  conditions:
  - lastTransitionTime: 2018-04-10T00:58:17Z
    message: 'no response from https://10.0.23.219:443: Get https://10.0.23.219:443:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Initially kubectl get --raw /apis/metrics.k8s.io works, but after a short time the resource becomes unavailable.

I'm running 1.9.6 on GKE.

@fengjian1585

This comment has been minimized.

fengjian1585 commented Apr 11, 2018

@groob
请问问题解决了吗?

@antcs

This comment has been minimized.

antcs commented Apr 17, 2018

I have the same issue as @sujithvs74 and @groob on my kubernetes cluster...

After the deployment, if I execute kubectl get apiservice v1beta1.metrics.k8s.io -o yaml I get following output:

status:
  conditions:
  - lastTransitionTime: 2018-04-17T12:34:34Z
    message: endpoints for service/metrics-server in "kube-system" have no addresses
    reason: MissingEndpoints
    status: "False"
    type: Available

After some while I get this status:

status:
  conditions:
  - lastTransitionTime: 2018-04-17T12:34:34Z
    message: 'no response from https://10.100.11.77:443: Get https://10.100.11.77:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

I checked the selector attribute from the metrics-server-service.yaml and searched for the pod with kubectl get pods --namespace=kube-system --selector=k8s-app=metrics-server.

The pod is running on my cluster and has following logs:

I0417 13:23:03.010262       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0417 13:23:03.010412       1 heapster.go:72] Metrics Server version v0.2.1
I0417 13:23:03.010642       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I0417 13:23:03.010656       1 configs.go:62] Using kubelet port 10255
I0417 13:23:03.011659       1 heapster.go:128] Starting with Metric Sink
I0417 13:23:03.531631       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0417 13:23:03.670048       1 heapster.go:101] Starting Heapster API server...
[restful] 2018/04/17 13:23:03 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/04/17 13:23:03 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0417 13:23:03.671412       1 serve.go:85] Serving securely on 0.0.0.0:443

Running on Kubernetes v1.9.3.

@antcs

This comment has been minimized.

antcs commented Apr 17, 2018

SOLUTION (if you are behind Corporate Proxy)

  1. Get Cluster-IP of your metrics server kubectl -n=kube-system get services
  2. Add the IP to no_proxy variable in the kube-apiserver config vi /etc/kubernetes/manifests/kube-apiserver.yaml
  3. reload systemctl daemon-reload && systemctl restart kubelet
  4. should work now kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
@swatisehgal

This comment has been minimized.

swatisehgal commented Apr 19, 2018

I am facing the same issue and getting the following error:
Error from server (ServiceUnavailable): the server is currently unable to handle the request
Tried the above suggestion but still stuck. Any more suggestions?

@DirectXMan12

This comment has been minimized.

Contributor

DirectXMan12 commented Apr 24, 2018

Whatever your cluster networking is, it needs to permit your API server to be able to talk to pods. Please ensure that this is the case, and ensure (as mentioned above) that your master isn't being routed through a proxy.

@yinwoods

This comment has been minimized.

yinwoods commented May 15, 2018

@antcs In 2 step, what do you mean no_proxy variable, I got the same problem here.

@psoares

This comment has been minimized.

psoares commented Aug 1, 2018

Same problem when deploying on Amazon EKS... and there I don't think I can edit apiserver.yaml.

@zknill

This comment has been minimized.

zknill commented Sep 14, 2018

FYI I fixed this problem on Amazon EKS by updating the kubernetes nodes security groups to allow ingress / incoming https connections from the EKS masters security group.

@craftyc0der

This comment has been minimized.

craftyc0der commented Sep 14, 2018

As a follow up to @zknill comment. For EKS to work, you must also allow 443 egress from the control plane to the nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment