Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics server api not getting registered #45

Closed
sujithvs74 opened this issue Mar 22, 2018 · 17 comments
Closed

Metrics server api not getting registered #45

sujithvs74 opened this issue Mar 22, 2018 · 17 comments
Labels

Comments

@sujithvs74
Copy link

@sujithvs74 sujithvs74 commented Mar 22, 2018

I have deployed metrics api in kubernetes following https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy/1.8%2B. Metric server is running fine and but I am not able to get metrics from it. I am using kubernetes 1.9 version.

[demo@dev-demo metrics-server]$ kubectl get --raw "/apis/metrics.k8s.io" Error from server (NotFound): the server could not find the requested resource [demo@dev-demo metrics-server]$

`[demo@dev-demo metrics-server]$ k get hpa -n default
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache / 50% 1 10 1 19h
[demo@dev-demo metrics-server]$ k describe hpa -n default
Name: php-apache
Namespace: default
Labels:
Annotations:
CreationTimestamp: Wed, 21 Mar 2018 04:57:32 -0400
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): / 50%
Min replicas: 1
Max replicas: 10
Conditions:
Type Status Reason Message


AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message


Warning FailedComputeMetricsReplicas 43m (x13 over 49m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 4m (x91 over 49m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)`

@DirectXMan12
Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Mar 23, 2018

if you do kubectl get apiservice v1beta1.metrics.k8s.io -o yaml, does the status look ok?

@groob
Copy link

@groob groob commented Apr 10, 2018

Same issue as @sujithvs74

status:
  conditions:
  - lastTransitionTime: 2018-04-10T00:58:17Z
    message: 'no response from https://10.0.23.219:443: Get https://10.0.23.219:443:
      net/http: request canceled (Client.Timeout exceeded while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

Initially kubectl get --raw /apis/metrics.k8s.io works, but after a short time the resource becomes unavailable.

I'm running 1.9.6 on GKE.

@fengjian1585
Copy link

@fengjian1585 fengjian1585 commented Apr 11, 2018

@groob
请问问题解决了吗?

@antcs
Copy link

@antcs antcs commented Apr 17, 2018

I have the same issue as @sujithvs74 and @groob on my kubernetes cluster...

After the deployment, if I execute kubectl get apiservice v1beta1.metrics.k8s.io -o yaml I get following output:

status:
  conditions:
  - lastTransitionTime: 2018-04-17T12:34:34Z
    message: endpoints for service/metrics-server in "kube-system" have no addresses
    reason: MissingEndpoints
    status: "False"
    type: Available

After some while I get this status:

status:
  conditions:
  - lastTransitionTime: 2018-04-17T12:34:34Z
    message: 'no response from https://10.100.11.77:443: Get https://10.100.11.77:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

I checked the selector attribute from the metrics-server-service.yaml and searched for the pod with kubectl get pods --namespace=kube-system --selector=k8s-app=metrics-server.

The pod is running on my cluster and has following logs:

I0417 13:23:03.010262       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0417 13:23:03.010412       1 heapster.go:72] Metrics Server version v0.2.1
I0417 13:23:03.010642       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I0417 13:23:03.010656       1 configs.go:62] Using kubelet port 10255
I0417 13:23:03.011659       1 heapster.go:128] Starting with Metric Sink
I0417 13:23:03.531631       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0417 13:23:03.670048       1 heapster.go:101] Starting Heapster API server...
[restful] 2018/04/17 13:23:03 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2018/04/17 13:23:03 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0417 13:23:03.671412       1 serve.go:85] Serving securely on 0.0.0.0:443

Running on Kubernetes v1.9.3.

@antcs
Copy link

@antcs antcs commented Apr 17, 2018

SOLUTION (if you are behind Corporate Proxy)

  1. Get Cluster-IP of your metrics server kubectl -n=kube-system get services
  2. Add the IP to no_proxy variable in the kube-apiserver config vi /etc/kubernetes/manifests/kube-apiserver.yaml
  3. reload systemctl daemon-reload && systemctl restart kubelet
  4. should work now kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
@swatisehgal
Copy link

@swatisehgal swatisehgal commented Apr 19, 2018

I am facing the same issue and getting the following error:
Error from server (ServiceUnavailable): the server is currently unable to handle the request
Tried the above suggestion but still stuck. Any more suggestions?

@DirectXMan12
Copy link
Contributor

@DirectXMan12 DirectXMan12 commented Apr 24, 2018

Whatever your cluster networking is, it needs to permit your API server to be able to talk to pods. Please ensure that this is the case, and ensure (as mentioned above) that your master isn't being routed through a proxy.

@yinwoods
Copy link

@yinwoods yinwoods commented May 15, 2018

@antcs In 2 step, what do you mean no_proxy variable, I got the same problem here.

@psoares
Copy link

@psoares psoares commented Aug 1, 2018

Same problem when deploying on Amazon EKS... and there I don't think I can edit apiserver.yaml.

@zknill
Copy link

@zknill zknill commented Sep 14, 2018

FYI I fixed this problem on Amazon EKS by updating the kubernetes nodes security groups to allow ingress / incoming https connections from the EKS masters security group.

@craftyc0der
Copy link

@craftyc0der craftyc0der commented Sep 14, 2018

As a follow up to @zknill comment. For EKS to work, you must also allow 443 egress from the control plane to the nodes.

@lolspider
Copy link

@lolspider lolspider commented Dec 6, 2018

i solve it by upgrade kubernetes to v1.12.3

@abdennour
Copy link

@abdennour abdennour commented Dec 20, 2018

@antcs please elaborate:

Add the IP to no_proxy variable in the kube-apiserver config vi /etc/kubernetes/manifests/kube-apiserver.yaml

can you give yaml snippet?

@gavinB-orange
Copy link

@gavinB-orange gavinB-orange commented Apr 4, 2019

@antcs solution worked for me - but only once I restarted the metrics-server pod

@kksharma1618
Copy link

@kksharma1618 kksharma1618 commented Jun 12, 2019

As a follow up to @zknill comment. For EKS to work, you must also allow 443 egress from the control plane to the nodes.

Thanks @zknill and @craftyc0der for this. Updating security groups on worker and control plane (both ingress and egress) worked :)

@vikrantsundriyal
Copy link

@vikrantsundriyal vikrantsundriyal commented Sep 6, 2019

zknill

FYI I fixed this problem on Amazon EKS by updating the kubernetes nodes security groups to allow ingress / incoming https connections from the EKS masters security group.

Thanks Buddy.. Your had a perfect reply for this problem coming on Amazon EKS. Following basic installations if still we get reason: FailedDiscoveryCheck and status: "False" and type: Available, its simply because of security group's inbound rule issue. Giving proper https access to Cluster Security group to access Node security group solves the problem.Thanks..

@mikhno-s
Copy link

@mikhno-s mikhno-s commented Oct 11, 2019

If you have the same problem on GKE cluster - you need to add prometheus-adapter port (not a kubernetes service port) to allowed in gcp firewall rules. By default kube-master is allowed to make requests only to 10250 and 443 ports. 6443 (default prom-adapter port) need to be added to this rule (or you can add another one with this port and filter by master ip)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.