aws-cloudwatch scaler only works with expressions that return a value greater than activationTargetMetricValue

[karl-sprig]

Proposal

I would like greater control over the logic in the IsActive() function for the aws-cloudwatch scaler.

For my needs (outlined below) there are a few different ways this could be implemented:

• just a simple way to indicate that the IsActive() logic should be negated. This could be a simple targetMetricComparisonOperator field which defaults to the current > but would also support <:

  # See: https://keda.sh/docs/2.8/scalers/aws-cloudwatch/
  - type: aws-cloudwatch
    <...>
    # We want keda to do $action when the value returned by CW api is < the target
    targetMetricComparisonOperator: "<"

• support for multiple expressions submitted to the cloudwatch API + a way to indicate which query should be used in the IsActive() calculation:

  triggers:
    - type: aws-cloudwatch
      metadata:
        expressions:
          # Cloudwatch does not support nested queries so we must assign each query an ID which CAN be referenced in other queries
          # First, get the total number of invocations
          # Then compare the results from m1 to the integer 1. If m1 >= 1, return 1. Else, 0.
          - id: m1
            expression: >-
              SELECT COUNT(Invocations) FROM SCHEMA("AWS/Lambda", FunctionName) WHERE FunctionName = 'sacred-cow'
          - id: e1
            expression: >-
              IF(m1 >=, 1, 1, 0)
          # Tell keda to watch only the value of the query identified as `e1` when making decisions about scaling
          targetMetricExpressionID: e1

• Implement something else entirely using a combination of Issue #2440 and Issue #3330

  • While both issues are worth perusing/implementing, they don't solve the more local problem: "the cloudwatch scaler only supports a single basic query expression"

TL;DR: Cloudwatch explicitly supports multiple queries, the keda scaler should too.

Use-Case

I have a sacred cow / legacy application that - from time to time - stop running properly.
For reasons that are not important here, this application is off limits.

Fortunately, I have some cloud watch metrics that allow me to see the number times this legacy application has been run per unit time. If the number drops below 1, I need to fire off a simple reboot script which - while blunt - does work reliably.

I have tried to implement this basic "if the count drops to less than 1, run the action" pattern with this:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
  <...>
spec:
  # Check every 60s
  pollingInterval: 60
  # At no point do we need more than 1 power-cycler instance running
  minReplicaCount: 0
  maxReplicaCount: 1

  triggers:
    # See: https://keda.sh/docs/2.8/scalers/aws-cloudwatch/
    - type: aws-cloudwatch
      metadata:
        namespace: AWS/Lambda
        metricName: Invocations
        expression: >-
          SELECT MIN(Invocations) FROM SCHEMA("AWS/Lambda", FunctionName, Resource) WHERE FunctionName = 'sacred-cow' AND Resource = 'sacred-cow'

        metricStatPeriod: "60"
        metricUnit: "Count"
        metricCollectionTime: "300"
        minMetricValue: "0"
        targetMetricValue: "1"
        activationTargetMetricValue: "0"

  jobTargetRef:
    <...>

However, after some conversation with @JorTurFer on slack, this may not be supported directly.

In short, the expression will always return 1 when things are nominal. Only when there is no data or the number of invocations per unit time is 0 will the expression return 0.

A quick 'hack' was suggested: Invert the logic; return 0 when things are nominal and 1 when things are not.

This is only possible when using logical operators in your cloudwatch query. However this requires submititng multiple queries to cloudwatch... which the current aws-cloudwatch scaler does not support.

To achieve the 'negate the expression result' workaround from above, you must submit an API query with a payload similar to:

{
    "metrics": [
        [ { "expression": "SELECT COUNT(Invocations) FROM SCHEMA(\"AWS/Lambda\", FunctionName) WHERE FunctionName = 'sacred-cow'", "label": "Metric1", "id": "mInvocationCount", "period": 60, "stat": "SampleCount" } ],
        [ { "expression": "IF(mInvocationCount > 0, 0, 1)", "label": "Expression1", "id": "e1" } ]
    ],
    "view": "timeSeries",
    "stacked": false,
    "region": "us-east-1",
    "stat": "SampleCount",
    "period": 60
}

However, the current implementation does not permit this; there is only ever one query expression sent to AWS:

keda/pkg/scalers/aws_cloudwatch_scaler.go

Line 349 in 96e5bd2

MetricDataQueries: []*cloudwatch.MetricDataQuery{

Anything else?

No response

[JorTurFer]

I answered you in slack but let's talk about this here to track the conversation.
why do you need 2 queries? why not just if(select count(invocations)>0, 1,0) ? I'm not an expert in AWS and maybe this is not doable...

I have seen that selecting the symbol used for the activation could be enough to you. AFAIK AWS CloudWatch supports arithmetic operations, just multiply you value for -1 and you have already changed the symbol in the upstream, any change is needed in KEDA side. Could it work?

[karl-sprig]

I'm not an expert in AWS and maybe this is not doable...

Not to worry. AWS does not explicitly spell this out anywhere that I can find. It's implied from all of the examples though. You can also confirm this experimentally as i'll show below.

Cloudwatch does not support nested queries in the form of Select SUM($thing) where $thing_id in (Select from ....)

AFAIK AWS CloudWatch supports arithmetic operations, just multiply you value for -1 and you have already changed the symbol in the upstream, any change is needed in KEDA side. Could it work?

Same with math operations. You get a single expression per query id and then you can compose the ids with operators but you can not use an operator and a 'normal' query in the same expression:

versus

I hope that clarifies things

[JorTurFer]

mmm, in that case, seems that ATM your request is not supported by KEDA, sorry.
Once formulas feature is ready this could work, but I don't have any idea for the moment as built-in :(

[JorTurFer]

one thing that you could do is create your own RestAPI/gRCP service which queries cloudwatch and applies the formulas that you need, and use Metrics API Scaler or External Scaler to query your service from KEDA
I guess that it will solve your problems with a few effort

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.