Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix HPA bug about unintentional scale out during updating deployment. #85027

Conversation

shibataka000
Copy link
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

During rolling update with maxSurge: 1 and maxUnavailable: 0, len(metrics) may be greater than currentReplcas and it may cause unintentional scale out in spite of usageRatio and newUsageRatio are less than 1.0. This PR fix that bug.

More detail scenario description and the way to reproduce bug is described in #84142 . Please see it.

Which issue(s) this PR fixes:

Fixes #84142

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 9, 2019
@k8s-ci-robot
Copy link
Contributor

Welcome @shibataka000!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @shibataka000. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 9, 2019
@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 9, 2019
During rolling update with maxSurge=1 and maxUnavailable=0,
len(metrics) is greater than currentReplcas
and it may cause unintentional scale out.
@shibataka000 shibataka000 force-pushed the fix-bug-about-unintentional-scale-out-during-updating-deployment branch from e4a0818 to b712277 Compare November 9, 2019 06:24
@neolit123
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 9, 2019
@shibataka000
Copy link
Contributor Author

/assign @piosz

@shibataka000
Copy link
Contributor Author

@piosz Could you review this PR?

@shibataka000
Copy link
Contributor Author

@piosz @mwielgus Could you review this PR?

@josephburnett
Copy link
Member

/lgtm
/approve

@shibataka000 can you apply this fix to calcPlainMetricReplicas as well here:

Thank you for this pull request!

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 24, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: josephburnett, shibataka000

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 24, 2020
@josephburnett
Copy link
Member

/test pull-kubernetes-e2e-gce

@k8s-ci-robot k8s-ci-robot merged commit c441a1a into kubernetes:master Mar 24, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Mar 24, 2020
@shibataka000 shibataka000 deleted the fix-bug-about-unintentional-scale-out-during-updating-deployment branch March 25, 2020 05:02
shibataka000 added a commit to shibataka000/kubernetes that referenced this pull request Mar 25, 2020
This commit fix bug in calcPlainMetricReplicas function.
Same bug in GetResourceReplicas function was fixed in kubernetes#85027.
@shibataka000
Copy link
Contributor Author

@josephburnett Thank you for your review!

can you apply this fix to calcPlainMetricReplicas as well here:

I did it in #89465 . Could you review it?

@josephburnett
Copy link
Member

@shibataka000 sure think, I will review once the automated tests finish.

k8s-ci-robot added a commit that referenced this pull request Apr 4, 2020
…-#85027-upstream-release-1.17

Automated cherry pick of #85027: Fix bug about unintentional scale out during updating
k8s-ci-robot added a commit that referenced this pull request Apr 4, 2020
…-#85027-upstream-release-1.16

Automated cherry pick of #85027: Fix bug about unintentional scale out during updating
k8s-ci-robot added a commit that referenced this pull request Apr 4, 2020
…-#85027-upstream-release-1.18

Automated cherry pick of #85027: Fix bug about unintentional scale out during updating
k8s-ci-robot added a commit that referenced this pull request Apr 9, 2020
…-#85027-upstream-release-1.15

Automated cherry pick of #85027: Fix bug about unintentional scale out during updating
hidetatz pushed a commit to hidetatz/kubernetes that referenced this pull request Jan 12, 2021
This commit fix bug in calcPlainMetricReplicas function.
Same bug in GetResourceReplicas function was fixed in kubernetes#85027.
hex108 pushed a commit to hex108/kubernetes that referenced this pull request Jan 13, 2021
This commit fix bug in calcPlainMetricReplicas function.
Same bug in GetResourceReplicas function was fixed in kubernetes#85027.
hex108 pushed a commit to hex108/kubernetes that referenced this pull request Jan 13, 2021
Fix bug about unintentional scale out during updating deployment.
**MR 描述 / 目的**:
Fix bug about unintentional scale out during updating deployment.

This commit fix bug in calcPlainMetricReplicas function.
Same bug in GetResourceReplicas function was fixed in kubernetes#85027.

**关联 issue**:
Fixes #

**代码审查须知**:

**MR 是否对用户有影响?**:

```发布须知

```
chenchun pushed a commit to chenchun/kubernetes that referenced this pull request Mar 20, 2024
This commit fix bug in calcPlainMetricReplicas function.
Same bug in GetResourceReplicas function was fixed in kubernetes#85027.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rolling update of deployments creates MAX number of pods X 2 according to HPA, regardless of current load
5 participants