Value of custom metric not being consumed properly

[bodymindarts]

I’m having issues trying to ingest custom metrics.
When sending this metric:

curl -Lvk $autoscaler_url/v1/apps/$app_uuid/metrics -H "Content-Type: application/json" --user $autoscaler_user:$autoscaler_password -X POST -d @- << EOF
{
  "instance_index": 0,
  "metrics": [
    {
      "name": "custom",
      "value": 1000,
      "unit": "test-unit"
    }
  ]
}
EOF

This is what shows up:

cf asm cf-env custom
Retrieving aggregated custom metrics for app cf-env...
Metrics Name            Value           Timestamp
custom                                  2020-06-29T13:23:29+02:00

The value is missing and no action is being taken even though the attached policy is:

{
  "instance_min_count": 1,
  "instance_max_count": 3,
  "scaling_rules": [
    {
      "metric_type": "custom",
      "threshold": 100,
      "operator": ">",
      "adjustment": "+1"
    }
  ]
}

Would appreciate any help figuring out why the "Value" is missing and what needs to be done to take action based on a custom metric.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/173558891

The labels on this github issue will be updated when the story is started.

[aqan213]

@bodymindarts I noticed that you opened another issue cloudfoundry/app-autoscaler-release#243 in app-autoscaler-release which is closed by yourself. In that issue the metricsforwarder was failed to send data to loggregator_agent due to the cert issue. Did it resolved?

And for this issue , does the custom metrics value show up now?

[bodymindarts]

We fixed the cert issue but this isn't resolved completely. The policy now activates correctly but the values still don't show up.

[aqan213]

The policy now activates correctly but the values still don't show up.

Do you mean your application can scale in/out correctly but when you run cf asm cf-env custom the value is empty?

Could you please check if the custom metrics is in the app_metric table?

select * from app_metric where app_id='<app_uuid>';

[bodymindarts]

Yes that's right it scaled in/out:

$ cf ash cf-env
Retrieving scaling event history for app cf-env...
Scaling Type            Status          Instance Changes        Time                            Action                                                                  Error
dynamic                 succeeded       3->2                    2020-06-29T21:01:55+02:00       -1 instance(s) because custom > 100test-unit for 120 seconds
dynamic                 succeeded       2->3                    2020-06-29T17:40:55+02:00       +1 instance(s) because custom > 100test-unit for 120 seconds
dynamic                 succeeded       1->2                    2020-06-29T17:12:55+02:00       +1 instance(s) because custom > 100test-unit for 120 seconds

But the values aren't being persisted or displayed:

                app_id                | metric_type  |   unit    |      timestamp      | value
--------------------------------------+--------------+-----------+---------------------+-------
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435677198003081 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435717175326841 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435757175106101 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435797172395151 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435837171363097 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435877172797696 |
 9aa474dc-7b6d-4cb1-bbf9-2ffb7d23c0d7 | custom       |           | 1593435917172673518 |

[aqan213]

I tried to send custom metrics with your curl command and the policy. I can get the metrics value:

cf asm nodeapp custom
Retrieving aggregated custom metrics for app nodeapp...
Metrics Name     	Value             	        Timestamp
custom           	1000test-unit     	2020-07-01T11:32:55+08:00
custom           	1000test-unit     	2020-07-01T11:32:15+08:00
custom           	1000test-unit     	2020-07-01T11:31:35+08:00
custom           	                  	       2020-07-01T11:30:55+08:00
custom           	                  	       2020-07-01T11:30:15+08:00
custom           	                  	       2020-07-01T11:29:35+08:00
custom           	                  	       2020-07-01T11:28:55+08:00
custom           	                  	       2020-07-01T11:28:15+08:00
custom           	                  	       2020-07-01T11:27:35+08:00
custom           	                  	      2020-07-01T11:26:55+08:00
custom           	                  	      2020-07-01T11:26:15+08:00
custom           	                  	      2020-07-01T11:25:35+08:00
custom           	                  	      2020-07-01T11:24:55+08:00

From the scaling history you posted above, the scaling action was before you queried from database.
The scaling action was happened at 2020-06-29T17:12:55+02:00 , 2020-06-29T17:40:55+02:00
but the queries are

timestamp                                   +0200                  
1593435677198003081          2020-06-29 15:01:17
1593435717175326841           2020-06-29 15:01:57
1593435757175106101
1593435797172395151
1593435837171363097
1593435877172797696
1593435917172673518           2020-06-29 15:05:17

Could you please check if there are metrics values in the database when the scaling action was taken?

select * from app_metric where app_id='<app_uuid>'  and timestamp >= $start_timestamp and timestamp <= $end_timestamp;

Or Could you please send the metrics again to re-trigger the scaling action and check if the values are displayed?

[bodymindarts]

I'm closing this because its no longer blocking the use case I was attempting to implement. I don't know what was wrong with the test I was attempting ^^^ but things are working as expected now.