[HPA e2e] Calculate more precise consumed CPU usage for N replicas

[pbeschetnov]

What type of PR is this?

/kind feature
/kind flake

What this PR does / why we need it:

With the current HPA e2e tests with behavior setup of:

• pod CPU request = 500m
• target CPU utilization = 25%
• usage for single replica = 110m

and

• usage intended for recommending n pods = n * {usage for single replica}

the consumed CPU per n replicas would be

replicas	recommendation = replicas * usage / request / target
1	0.88
2	1.76
3	2.64
4	3.52
5	4.4
6	5.28
7	6.16
8	7.04
9	7.92
10	8.8

As shown, the usage is more balanced for 4-5 replicas, and skewed for <=3 and >=6 replicas — this leads to flaky tests when consumed CPU by the ResourceConsumer fluctuates and goes over the intended usage. Example:

1. Requesting usage for 2 replicas.
2. RC consumer: sending request to consume 220 millicores.
3. Actual usage: 254 millicores.
4. The recommendation: 254 / 500 / 0.25 = 2.032 — rounding up to 3.
5. Here we correctly recommended 3, but the test setup expected 2, so the test failed.

For >=9 replicas it even produces not enough usage for the intended recommendation.

Instead of using this approach, I suggest calculating the replica usage for n pods as if n - 0.5 replicas consume all CPU matching the target. It would be:

(replicas - 0.5) * request * targetPercentage / 100%

The 0.5 replica reduction is to accommodate for the deviation between the actual consumed cpu and requested usage by the ResourceConsumer. HPA rounds up the recommendations. So, if the usage is e.g. for 3.5 replicas, the recommended replica number will be 4.

This will eliminate flakiness in HPA e2e tests with behavior.

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Hi @pbeschetnov. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pbetkier]

/ok-to-test
/lgtm

nice idea 👍 , let's see how it helps with flakes

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 2fb282de5b80d93f66739dd93f4e15bae8241f41

[pbeschetnov]

pull-kubernetes-e2e-autoscaling-hpa-cpu logs are gone
/retest

[pbeschetnov]

/assign @mwielgus

[mwielgus]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mwielgus, pbeschetnov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/e2e/autoscaling/OWNERS [mwielgus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment