Fix HPA settings to make them more reactive

[L3n41c]

What does this PR do?

• In the spec of HorizontalPodAutoscaler, use Value instead of AverageValue because the query in the DatadogMetric is already computing an average (avg: operator at the beginning of the query).
  As a result, the total number of TPS was wrongly divided twice by the number of pods (once by datadog because of the avg operator in the query and once by the HPA itself)
• Reduce the stabilization windows because:

  • We want the HPA to be as reactive as possible in the context of e2e to detect scale up and scale down events as fast as possible here.

  • The doc says:

    The stabilization window is used to restrict the flapping of replica count when the metrics used for scaling keep fluctuating.

    We don’t need it because the query in the DatadogMetric already smoothes the values with the .rollup(60).

Which scenarios this will impact?

• aws/eks
• aws/kindvm

Motivation

We noticed some flaky failures of the testHPA in the past.

Additional Notes

This graph shows the behavior of the HPA before and after this change.

[L3n41c]

/merge

[dd-devflow]

🚂 MergeQueue

Pull request added to the queue.

This build is going to start soon! (estimated merge in less than 22m)

Use /merge -c to cancel this operation!