New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pod hpa would create extra pods during deployment rolling update #72775

Open
zq-david-wang opened this Issue Jan 10, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@zq-david-wang
Copy link

zq-david-wang commented Jan 10, 2019

What happened:
I have deployment with following upgrade strategy

strategy:
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
  type: RollingUpdate

and hpa:

spec:
maxReplicas: 3
minReplicas: 1
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: xxxx
targetCPUUtilizationPercentage: 80

When sending patch command to update container image, (current replicas is 1), following happended

  1. new pod created and getting ready
  2. after the new pod ready, the old pod is terminating.
  3. hpa make a judgement call to set desired replica to 2
  4. after a while, hpa readjust to desired replica back to 1

What you expected to happen:

  1. new pod created and getting ready
  2. after the new pod ready, the old pod is terminating.

How to reproduce it (as minimally and precisely as possible):

  1. create a deployment with replica set to 1, maxSurge 1, maxUnavailable: 0
  2. create a hpa targeting the deployment with replica 1 from 3, 80% cpu target
  3. change container image via kubectl edit or other approach
  4. keep checking pod count changes

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@zq-david-wang

This comment has been minimized.

Copy link

zq-david-wang commented Jan 10, 2019

@kubernetes/sig-autoscalling

@zq-david-wang

This comment has been minimized.

Copy link

zq-david-wang commented Jan 10, 2019

/sig autoscalling

@zq-david-wang

This comment has been minimized.

Copy link

zq-david-wang commented Jan 11, 2019

It seems that there is several issue in hpa code, the calculation is based on current replica, but during rolling upgrade, the max surge would create extra replica containing different versions of replica set, but after calculation the desired replica is written all to the current version of replica set via deployment spec

@zq-david-wang

This comment has been minimized.

Copy link

zq-david-wang commented Jan 11, 2019

/sig autoscaling

@zq-david-wang zq-david-wang referenced a pull request that will close this issue Jan 11, 2019

Open

skip hpa when rolling update is in progress #72814

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment