CreateNewMetricsCollector : Failed to create new metrics collector

[humayunjamal]

Expected Behavior

external and custom metrics api should give some output but there is none

Actual Behavior

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "external.metrics.k8s.io/v1beta1",
"resources": []
}

Steps to Reproduce the Problem

1. Using the manifests with some updation of config parameters in (https://github.com/zalando-incubator/kubernetes-on-aws/tree/dev/cluster/manifests/kube-metrics-adapter), I am able to run the kube-metrics-apdapter but the logs of the container constantly show the below warnings

time="2019-01-22T10:32:24Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-01-22T10:32:24Z" level=info msg="Found 1 new/updated HPA(s)" provider=hpa
time="2019-01-22T10:32:24Z" level=info msg="Event(v1.ObjectReference{Kind:\"HorizontalPodAutoscaler\", Namespace:\"default\", Name:\"myservice-k8\", UID:\"ba31da13-1e10-11e9-a841-06f3d10b141c\", APIVersion:\"autoscaling/v2beta1\", ResourceVersion:\"36437\", FieldPath:\"\"}): type: 'Warning' reason: 'CreateNewMetricsCollector' Failed to create new metrics collector: format '' not supported"

Can not find any documentation or examples explaining what is expected here ?

Using the example I am able to successfully run kube-ingress-aws-controller in (https://github.com/zalando-incubator/kube-ingress-aws-controller/tree/master/deploy)

And my prometheus stack detects the skipper metrics and I can see in my promethues graphs the values such as

skipper_response_duration_seconds_sum{application="skipper",code="200",component="ingress",controller_revision_hash="1183544327",instance="172.20.4.24:9911",job="kubernetes-pods",kubernetes_namespace="kube-system",kubernetes_pod_name="skipper-ingress-mpm6g",method="GET",pod_template_generation="3",route="kube_default__myservice_k8__myservice_k8_stag_mydomain_com____myservice_k8_0__lb_group"}

Now I am expecting/assuming that when I run kube-metrics-adapter with configurations mentioned above , I should be able to see some metrics under api /apis/external.metrics.k8s.io/v1beta1 or /apis/custom.metrics.k8s.io/v1beta1
but its empty .

Remember myservice is NOT running any metrics exporter.

Specifications

• Version:1.11
• Platform: EKS
• Subsystem:

[mikkeloscar]

What does your hpa resource look like? I think it's not configured as expected by the metrics-adapter which is why you see the log: Failed to create new metrics collector: format '' not supported

[humayunjamal]

@mikkeloscar the hpa setting for myservice is

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myservice-k8
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: myservice-k8
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 1G
  - type: Pods
    pods:
      metricName: cpu_system
      targetAverageValue: 20m

[humayunjamal]

expected by the metrics-adapter

Also please escuse my ignorance where exactly I can find the expectations of the metrics-adapter in HPA. the json examples or documentation given in the main readme is not clear enough or detailed enough :(

[mikkeloscar]

Also please escuse my ignorance where exactly I can find the expectations of the metrics-adapter in HPA. the json examples or documentation given in the main readme is not clear enough or detailed enough :(

Sorry, we have tried to explain this in the README, please let us know if you have ideas on how to make this more clear?

For your HPA, the problem is that you specify metricName: cpu_system but the metrics-adapter doesn't know what to do with this metric. You need to also provide the annotations similar to what's shown in this example: https://github.com/zalando-incubator/kube-metrics-adapter#pod-collector

[humayunjamal]

@mikkeloscar
Do I need to run an exporter in my app ? which exports the metrics via /metrics plus it should be exposed in json format ?

[mikkeloscar]

Yes, that's what's currently supported for the Pods resource type in the metrics-adapter.

If you have the data in prometheus you could also use the prometheus collector by configuring your HPA like in this example: https://github.com/zalando-incubator/kube-metrics-adapter#prometheus-collector
but without knowing your application I'm not sure what you are trying to do and if this would make sense for you :)

[humayunjamal]

If you guys dont mind then can we have a full end to end example or walkthrough so people can quickly try it out and see if its the right tool for them or not. Because for me it seems to be an overkill as I will need to convert the exporters to write json output and then actually see how it benefits me. Just trying it out first time with the current README is unfortunately not detailed enough.

[szuecs]

@humayunjamal what do you want to achieve with the custom HPA? Scale by request per second or something App specific?

[humayunjamal]

@humayunjamal what do you want to achieve with the custom HPA? Scale by request per second or something App specific?

@szuecs Yes I want to achieve scaling by requests per second . Unfortunately the skipper /metrics also dont expose requests per second metrics.

[szuecs]

@humayunjamal as far as I remember we use the sum of all skipper instances 1m.rate divided by 60, so more sliding window style req/s. @arjunrn maybe you can tell us a bit about how to use it?

I guess you also do not really want to have the spiky requests per second in most cases, which would scale up and down way too fast and the hpa would be always in backoff state to calm down.

[arjunrn]

@humayunjamal The setup for the kube-metrics-adapter with skipper metrics is not explained well enough. Basically the skipper metrics are collected by prometheus and then prometheus is queried to provide aggregate numbers. I will add a How-To document over this weekend and hopefully you will be able to try it after that.

[mikkeloscar]

Closing as this was documented. Please re-open if something is still unclear!