Cant retrieve Custom Metrics #77

balamuru · 2019-08-22T17:06:14Z

Expected Behavior

Custom metrics exposed by spring boot actuator metric is read

Actual Behavior

Unable to get metrics

Steps to Reproduce the Problem

Deploy spring boot application in k8s using the dockerhub image https://cloud.docker.com/repository/docker/vinaybalamuru/spring-boot-hpa

kubectl run springboot-webapp --image=vinaybalamuru/spring-boot-hpa  --requests=cpu=200m --limits=cpu=500m --expose --port=7070

Install kube metrics adapter (custom and external etc). The adapter currently works for external Prometheus queries

sh-4.2$ kubectl get apiservices |grep metrics
v1beta1.custom.metrics.k8s.io          kube-system/kube-metrics-adapter   True        18h
v1beta1.external.metrics.k8s.io        kube-system/kube-metrics-adapter   True        5d23h
v1beta1.metrics.k8s.io                 kube-system/metrics-server         True        9d
sh-4.2$ kubectl get po -n kube-system |grep metrics
kube-metrics-adapter-55cbb64dc9-g22pb   1/1     Running   0          3d
metrics-server-9cb648b76-4j6tf          1/1     Running   2          9d

Create a HPA associated with springboot-webapp and intended to scale on metric load per minute

cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: springboot-custom-hpa
  namespace: default
  labels:
    application: custom-metrics-consumer
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.load-per-min.json-path/json-key: "$.measurements[:1].value"
    metric-config.pods.load-per-min.json-path/path: /actuator/metrics/system.load.average.1m
    metric-config.pods.load-per-min.json-path/port: "7070"
    metric-config.pods.load-per-min.json-path/scheme: "http"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: springboot-webapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: load-per-min
      targetAverageValue: 1
EOF

sh-4.2$ kubectl get hpa                  
NAME                                     REFERENCE                      TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
springboot-custom-hpa                    Deployment/springboot-webapp   <unknown>/1         1         10        2          18h
sh-4.2$

Describing the HPA. it looks like the HPA couldn't read the custom metrics

$ kubectl describe hpa springboot-custom-hpa 
Name:                      springboot-custom-hpa
Namespace:                 default
Labels:                    application=custom-metrics-consumer
Annotations:               kubectl.kubernetes.io/last-applied-configuration:
                             {"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{"metric-config.pods.load-per-min.json-path...
                           metric-config.pods.load-per-min.json-path/json-key: $.measurements[:1].value
                           metric-config.pods.load-per-min.json-path/path: /actuator/metrics/system.load.average.1m
                           metric-config.pods.load-per-min.json-path/port: 7070
                           metric-config.pods.load-per-min.json-path/scheme: http
CreationTimestamp:         Wed, 21 Aug 2019 17:03:16 -0500
Reference:                 Deployment/springboot-webapp
Metrics:                   ( current / target )
  "load-per-min" on pods:  <unknown> / 1
Min replicas:              1
Max replicas:              10
Deployment pods:           2 current / 2 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive   False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric load-per-min: no metrics returned from custom metrics API
  ScalingLimited  True    TooFewReplicas       the desired replica count is increasing faster than the maximum scale rate
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetPodsMetric           53m (x9 over 55m)    horizontal-pod-autoscaler  unable to get metric queue-length: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  53m (x9 over 55m)    horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get object metric value: unable to get metric queue-length: no metrics returned from custom metrics API
  Warning  FailedComputeMetricsReplicas  53m (x3 over 53m)    horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get object metric value: unable to get metric load-per-min: no metrics returned from custom metrics API
  Warning  FailedGetPodsMetric           54s (x209 over 53m)  horizontal-pod-autoscaler  unable to get metric load-per-min: no metrics returned from custom metrics API

I know that I can get into a pod and retrieve metrics from the service endpoint so I'm not sure where things are going wrong
eg

/ # wget -q -O- http://springboot-webapp.default.svc.cluster.local:7070/actuator/metrics/system.load.average.1m
{
  "name" : "system.load.average.1m",
  "description" : "The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time",
  "baseUnit" : null,
  "measurements" : [ {
    "statistic" : "VALUE",
    "value" : 0.0439453125
  } ],
  "availableTags" : [ ]
}/ #

Specifications

Version:

  kube-metrics-adapter:
    Container ID:  docker://5b32f88953acbb290a22279899d10f853dabc1d7d9a0dd582c9652e32ee61295
    Image:         registry.opensource.zalan.do/teapot/kube-metrics-adapter:latest
    Image ID:      docker-pullable://registry.opensource.zalan.do/teapot/kube-metrics-adapter@sha256:12bd1e57c8448ed935a876959b143827b1f8f070d7

Platform:
K8s 1.15
Subsystem:

The text was updated successfully, but these errors were encountered:

mikkeloscar · 2019-08-23T07:04:36Z

Thanks for a detailed report, could you also show the logs of the kube-metrics-adapter pod. I would imagine it would log some kind of error when trying to pull the metrics.

balamuru · 2019-08-23T14:53:19Z

Here you go . Looks like some sort out of range error

sh-4.2$ kubectl get hpa
NAME                    REFERENCE                      TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
springboot-custom-hpa   Deployment/springboot-webapp   <unknown>/1   1         10        2          40h
sh-4.2$ kubectl get po -n kube-system |grep metrics
kube-metrics-adapter-55cbb64dc9-g22pb   1/1     Running   0          3d22h
metrics-server-9cb648b76-4j6tf          1/1     Running   2          10d
sh-4.2$ kubectl logs  kube-metrics-adapter-55cbb64dc9-g22pb -n kube-system
.
.
.
time="2019-08-23T14:50:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-gc6pm': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:50:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-tdvkr': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:50:06Z" level=info msg="Collected 0 new metric(s)" provider=hpa
time="2019-08-23T14:50:08Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:50:08Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa
time="2019-08-23T14:50:38Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:50:38Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa
time="2019-08-23T14:51:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-gc6pm': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:51:06Z" level=error msg="Failed to get metrics from pod 'default/springboot-webapp-79475bc7c-tdvkr': index [to] out of range: len: 1, to: 1" Collector=Pod
time="2019-08-23T14:51:06Z" level=info msg="Collected 0 new metric(s)" provider=hpa
time="2019-08-23T14:51:08Z" level=info msg="Looking for HPAs" provider=hpa
time="2019-08-23T14:51:08Z" level=info msg="Found 0 new/updated HPA(s)" provider=hpa

mikkeloscar · 2019-08-23T15:03:12Z

Then the question is if this query $.measurements[:1].value is valid in https://github.com/oliveagle/jsonpath

At least it would not be so hard to test this with all the data you provided :)

mikkeloscar · 2019-08-23T15:07:57Z

I tested it like this and it is not valid at least with the json defined:

package main

import (
	"encoding/json"
	"fmt"

	"github.com/oliveagle/jsonpath"
)

const v = `{
  "name" : "system.load.average.1m",
  "description" : "The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time",
  "baseUnit" : null,
  "measurements" : [ {
    "statistic" : "VALUE",
    "value" : 0.0439453125
  } ],
  "availableTags" : [ ]
}`

func main() {
	var json_data interface{}
	json.Unmarshal([]byte(v), &json_data)

	res, err := jsonpath.JsonPathLookup(json_data, "$.measurements[:1].value")
	fmt.Println(err)
	fmt.Println(res)
}

Output is:

index [to] out of range: len: 1, to: 1
<nil>

Using something like $.measurements[:].value does work though, but I don't know if it represents the thing you want?

mikkeloscar · 2019-08-23T15:12:33Z

Using something like $.measurements[:].value does work though, but I don't know if it represents the thing you want?

However this does not work with the kube-metrics-adapter because we expect a single value to be the result rather than a list. I think you need to do: $.measurements[0].value to get something that makes sense.

balamuru · 2019-08-23T18:44:56Z

You are correct. It should be $.measurements[0].value. Custom autoscaling now works
Thanks
The valid command / yaml is

cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: springboot-custom-hpa
  namespace: default
  labels:
    application: custom-metrics-consumer
  annotations:
    # metric-config.<metricType>.<metricName>.<collectorName>/<configKey>
    metric-config.pods.load-per-min.json-path/json-key: "$.measurements[0].value"
    metric-config.pods.load-per-min.json-path/path: /actuator/metrics/system.load.average.1m
    metric-config.pods.load-per-min.json-path/port: "7070"
    metric-config.pods.load-per-min.json-path/scheme: "http"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: springboot-webapp
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: load-per-min
      targetAverageValue: 1
EOF

Sample output

$ kubectl get hpa --watch
NAME                    REFERENCE                      TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
springboot-custom-hpa   Deployment/springboot-webapp   189m/1    1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   435m/1    1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   1351m/1   1         10        1          44h
springboot-custom-hpa   Deployment/springboot-webapp   1351m/1   1         10        2          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        2          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        4          44h
springboot-custom-hpa   Deployment/springboot-webapp   2388m/1   1         10        8          44h
springboot-custom-hpa   Deployment/springboot-webapp   2644m/1   1         10        8          44h
springboot-custom-hpa   Deployment/springboot-webapp   2644m/1   1         10        10         44h
`

balamuru closed this as completed Aug 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cant retrieve Custom Metrics #77

Cant retrieve Custom Metrics #77

balamuru commented Aug 22, 2019

mikkeloscar commented Aug 23, 2019

balamuru commented Aug 23, 2019 •

edited

Loading

mikkeloscar commented Aug 23, 2019

mikkeloscar commented Aug 23, 2019 •

edited

Loading

mikkeloscar commented Aug 23, 2019

balamuru commented Aug 23, 2019

Cant retrieve Custom Metrics #77

Cant retrieve Custom Metrics #77

Comments

balamuru commented Aug 22, 2019

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

mikkeloscar commented Aug 23, 2019

balamuru commented Aug 23, 2019 • edited Loading

mikkeloscar commented Aug 23, 2019

mikkeloscar commented Aug 23, 2019 • edited Loading

mikkeloscar commented Aug 23, 2019

balamuru commented Aug 23, 2019

balamuru commented Aug 23, 2019 •

edited

Loading

mikkeloscar commented Aug 23, 2019 •

edited

Loading