Custom Metrics scaling: HPA did not receive metrics for any ready pods

[bibinwilson]

hi,

I am trying the custom metric autocaling as mentioned in https://github.com/kubeless/kubeless/tree/master/manifests/autoscaling

Kubernetes Env: GKE 1.8.4 with alpha features enabled

I am getting error in HPA. Here is output of HPA.

Name:                                                                sample-tomcat-app-hpa
Namespace:                                                           default
Labels:                                                              
Annotations:                                                         
CreationTimestamp:                                                   Wed, 13 Dec 2017 20:06:22 +0530
Reference:                                                           Deployment/sample-tomcat-app
Metrics:                                                             ( current / target )
  "tomcat_threadpool_connectioncount" on pods:                       unknown / 2
  "tomcat_threadpool_connectioncount" on Service/sample-tomcat-app:  unknown / 2
  resource cpu on pods  (as a percentage of request):                unknown / 50%
Min replicas:                                                        2
Max replicas:                                                        10
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: did not receive metrics for any ready pods
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedGetPodsMetric           14s (x8 over 3m)  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  14s (x8 over 3m)  horizontal-pod-autoscaler  failed to get pods metric value: did not receive metrics for any ready pods

Here is the list of enabled API's

apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1

I tried the sample metrics app as well..I was getting the similiar error. Any ideas why it is happening?

[sebgoa]

honestly we have never tried the autoscaling on GKE. There might be some additional configuration of the controller-manager that are requried:

https://github.com/kubeless/kubeless/tree/master/manifests/autoscaling

we need to investigate. if you could help that would be awesome.

[bibinwilson]

I am not sure if --horizontal-pod-autoscaler-use-rest-clients=true is set in the controller manager. As GKE is a managed kubernetes cluster, we do not have an option to edit the master configurations. I will post this in office GKE support forum and will get back to you.

[sebgoa]

thanks, I will also ping google :)

[ngtuna]

@bibinwilson Yes that would be great to check out if GKE support custom configuration on the controller-manager. I am not sure about that. Can you try this query just to be sure that the apigroup custom.metrics.k8s.io/v1beta1 works properly on your setup ?

$ kubectl get --raw /apis/custom-metrics.metrics.k8s.io/v1beta1

Also, this guy is trying GKE as well #488 . It's great that you guys can connect and test GKE. Your feedbacks are really valuable. Thanks 👍

[bibinwilson]

sure @ngtuna . I am working on it..If I get it to work, I will post my findings here. I will collaborate with @prakashsingh08.

[ngtuna]

@bibinwilson Do you have any update on this ?

[bibinwilson]

@ngtuna I spoke to google support team. They said this functionality will be added in GKE for 1.9 release..For 1.8x, we cannot implement custom metrics scaling using HPA.

[ngtuna]

Oh okay. Thank you for that info 👍 I guess in GKE 1.9 they will open options to configure the master node, one of them will be configuring the controller-manager.

[ngtuna]

So I guess we can close this issue. Will try it out when GKE 1.9 is available.

[YasminBZ]

Hello,

i tried another time to configure the autoscaling , i installed kubernetes 1.7 .
here is the output of the api-versions command :

admissionregistration.k8s.io/v1alpha1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2alpha1
batch/v1
batch/v2alpha1
certificates.k8s.io/v1beta1
extensions/v1beta1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1alpha1
rbac.authorization.k8s.io/v1beta1
settings.k8s.io/v1alpha1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
I can't see custom-metrics.metrics.k8s.io/v1alpha1 in the list !!!!
2 here is what i get as an error as well when executing this command:
kubectl create -f sample-metrics-app.yaml
deployment "sample-metrics-app" created
service "sample-metrics-app" created
horizontalpodautoscaler "sample-metrics-app-hpa" created
error: unable to recognize "sample-metrics-app.yaml": no matches for monitoring.coreos.com/, Kind=ServiceMonitor

Please can anyone help ?!
Thank you .

[ngtuna]

Hi @White-Cloud , what happens when you run this command:

$ kubectl create -f $KUBELESS_REPO/manifests/autoscaling/custom-metrics.yaml

[yiqu]

For me, I was able to solve this by setting --horizontal-pod-autoscaler-use-rest-clients=true in controller-manager, while having Heapster and metrics-server running as well.