Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple metrics hpa #78503

Merged
merged 2 commits into from Jun 4, 2019

Conversation

@gjtempleton
Copy link
Contributor

commented May 29, 2019

What type of PR is this?

/kind cleanup

What this PR does / why we need it:
Currently the HPA controller will fail to scale up targets if they are scaling on multiple metrics and one or more of these metrics is invalid/unavailable, even if one of the metrics is available and indicates a scale up should happen.

This PR allows a scale up to happen in this case, whilst still ensuring that it fails safe and refuses to scale down if one or more metrics are unavailable.

Which issue(s) this PR fixes:

Fixes #61007

Special notes for your reviewer:
I wasn't 100% sure whether to go for the first or last invalid metric to be reported as the error in the case of refusing to scale up, however going with the last simplified the changes required massively, meaning no need for passing around conditions.

Credit to @bskiba as a large amount of the changes come from her previous PR #61423

Does this PR introduce a user-facing change?:

Horizontal Pod Autoscaling can now scale targets up even when one or more metrics are invalid/unavailable as long as one metric indicates a scale up should occur.

/sig autoscaling
/priority important-longterm

gjtempleton added some commits May 29, 2019

Add tests for handling scaling on unavailable metrics
Add three tests for handling invalid metrics when scaling on
multiple metrics - one for scaling up successfully (new behaviour)
and two for ensuring we don't scale down (existing behaviour).
Handle invalid metrics when scaling on multiple metrics
Handle a case in the Horizontal Pod Autoscaler Controller when scaling
on multiple metrics and one or more is missing or invalid.

If all metrics are missing - return an error and leave the isScalingActive
condition as that for the last invalid metric.

If some metrics are missing/invalid and some are valid and found -
if a scale up would be triggered by the valid metrics ignore the missing
metrics and scale up, if a scale down would be triggered, return an error
and leave the isScalingActive condition as that for the last invalid metric.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 29, 2019

Hi @gjtempleton. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@gjtempleton

This comment has been minimized.

Copy link
Contributor Author

commented May 29, 2019

@josephburnett

This comment has been minimized.

Copy link

commented May 30, 2019

/lgtm

I'm not yet part of the Kubernetes org, so I can't add /ok-to-test. @mwielgus can you please.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 30, 2019

@josephburnett: changing LGTM is restricted to assignees, and only kubernetes/kubernetes repo collaborators may be assigned issues.

In response to this:

/lgtm

I'm not yet part of the Kubernetes org, so I can't add /ok-to-test. @mwielgus can you please.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@bskiba

This comment has been minimized.

Copy link
Member

commented May 30, 2019

/ok-to-test

@bskiba

This comment has been minimized.

Copy link
Member

commented May 31, 2019

Thanks!
/lgtm

@bskiba

This comment has been minimized.

Copy link
Member

commented May 31, 2019

/test pull-kubernetes-bazel-build

@bskiba

This comment has been minimized.

Copy link
Member

commented May 31, 2019

@MaciekPytel @mwielgus Can one of you approve?

@gjtempleton

This comment has been minimized.

Copy link
Contributor Author

commented May 31, 2019

/test pull-kubernetes-bazel-build

@bskiba

This comment has been minimized.

Copy link
Member

commented May 31, 2019

/assign @mwielgus
/assgin @MaciekPytel

@bskiba

This comment has been minimized.

Copy link
Member

commented May 31, 2019

/assign @MaciekPytel

@mwielgus
Copy link
Contributor

left a comment

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gjtempleton, mwielgus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mwielgus mwielgus added this to the v1.15 milestone Jun 3, 2019

@k8s-ci-robot k8s-ci-robot merged commit 9f85c5c into kubernetes:master Jun 4, 2019

21 checks passed

cla/linuxfoundation gjtempleton authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

@gjtempleton gjtempleton deleted the gjtempleton:Multiple-Metrics-HPA branch Jun 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.