New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In-place rolling updates #9043

Open
davidopp opened this Issue May 31, 2015 · 39 comments

Comments

@davidopp
Copy link
Member

davidopp commented May 31, 2015

Our current rolling update scheme requires deleting a pod and creating a replacement. We should also have rolling "in-place" update, of both containers and pods. There are two kinds of in-place update

  1. [only applies to container update] pod stays in place, but container doesn't - for example, change container image without killing pod
  2. [applies to pod or container update] pod stays in place, and container stays in place - for example, resource limit updates (IIRC we don't implement this yet, or at least don't implement it in-place)

The motivation is that there's no reason to kill the pod if you don't need to. The user may have local data stored in it that they don't want blown away due to update.

@erictune erictune added this to the v1.0-post milestone Jun 1, 2015

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Jun 1, 2015

I agree we'll eventually want this, but...

There are very few in-place, non-disruptive updates that we can actually do right now. For instance, Docker doesn't support resource changes, so we'll need to work around that using cgroup_parent. Resource changes will also complicate admission control in kubelet.

Decoupling lifetime of local storage from the pod is discussed in #7562 and #598, which would reduce the number of circumstances where in-place updates would be required.

@soundofjw

This comment has been minimized.

Copy link

soundofjw commented Jul 6, 2015

This would be wildly valuable - specifically for kubernetes + elastic-search, where data nodes really shouldn't be losing data.

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Jul 6, 2015

Data needs a lifetime independent of pods. See, for example, #598 and #7562.

@bgrant0607 bgrant0607 removed this from the v1.0-post milestone Jul 24, 2015

@bgrant0607 bgrant0607 added team/ux and removed sig/api-machinery labels Sep 2, 2015

@hurf

This comment has been minimized.

Copy link
Contributor

hurf commented Sep 15, 2015

It seems #6099 is also talking about this.

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Dec 4, 2015

@hurf #6099 is not related to this. That's nodes, this is pods.

@chengyli

This comment has been minimized.

Copy link
Contributor

chengyli commented Jan 19, 2016

What's the difference between this proposal and "kubectl replace/patch"?

@alfred-huangjian

This comment has been minimized.

Copy link
Contributor

alfred-huangjian commented Feb 1, 2016

@kubernetes/huawei

@davidopp

This comment has been minimized.

Copy link
Member Author

davidopp commented Feb 5, 2016

https://blog.docker.com/2016/02/docker-1-10/
(Docker 1.10 announcement)

Live update container resource constraints: When setting limits on what resources containers can use (e.g. memory usage), you had to restart the container to change them. You can now update these resource constraints on the fly with the new docker update command.

@davidopp

This comment has been minimized.

Copy link
Member Author

davidopp commented May 21, 2016

/inplaceupdate subresource on pod might be a good way to express pod updates (resource as well as container image)

@dts

This comment has been minimized.

Copy link

dts commented May 25, 2016

Mentioned in #13488 is an important use case for me: updating secrets. Specifically if we have a load balancer referencing certificates in a Secret and we want to refresh those certificates in a rolling fashion to all pods in the deployment/etc.

@mqliang

This comment has been minimized.

Copy link
Member

mqliang commented Jun 13, 2016

@davidopp May I ask what the benefit can we get from in-place rolling update? Is it about scheduling? IIRC, if we update pod.spec, kubelet will sync the pod. So, what we really need is just updating the spec of a running pod, kubelet will watched the change and do all the remaining task. Then, why do we need /inplaceupdate subresource on pod? Is it because /inplaceupdate will update the spec as well as the status, so we can wait kubelet sync the new spec and report the latest status?

@resouer

This comment has been minimized.

Copy link
Member

resouer commented Dec 21, 2016

Got a resource in the fly update request from #39060, which is mainly inspired by docker update.

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Feb 24, 2017

Pod resource updates is #5774

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Dec 21, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Jan 20, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Jan 22, 2018

/remove-lifecycle rotten
/lifecycle frozen

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Jun 2, 2018

Example: change to affinity: #57181

@fabiand

This comment has been minimized.

Copy link
Contributor

fabiand commented Jun 11, 2018

Considering the original 2nd case from the initial description - aka changing resources of a pod.

Assuming that we do not want to change the pod (and scheduler) behavior, I wonder if this could be modeled with some kind of PodResourceAttachment.

A resource attachment could contain additional memory, or cpu shares, eventually volmes (not sure how the api would look like for this), but if such a "attachment" is created on the cluster, then it can be applied to existing pods based on labels and selectors. The scheduler can check the constratins and either attach such an attachment to a pod or not, effectively granting additional resource to the pod.

In such a fashion the pod itself does not have to be changed, but it could be helpful for i.e. StatefulSet pods, which are running for some longer time.

@vinaykul

This comment has been minimized.

Copy link

vinaykul commented Sep 7, 2018

Hello,

Based on customer request, we have been looking into adding restart-free (in-place) vertical resources scaling feature to kubernetes for k8s Jobs. We reviewed VPA, looked at some of the earlier efforts (this thread for example), and worked on a proof-of-concept implementation that performs vertical resources scaling.

In our quick POC implementation, with minimally invasive changes we are able to update cpu/memory requests and limits for running pods by having the scheduler update its cache, and then have kubelet pick up the change and leverage UpdateContainerResources API for setting the updated limits without container restarts. We believe this approach fits well with the intrinsic flow of events from APIserver to scheduler to kubelet. Our next steps is to investigate extending VPA to leverage this feature in “Auto” mode, and seek guidance from VPA experts.

This feels like a good point to get feedback and guidance from k8s experts here, before we get too deep into VPA. Can you please review the following design document?
Best Effort In-Place Vertical Scaling

We are looking forward to your input, and to working with the community to get this upstream if this approach looks good to you.

Thanks
Vinay

CC: @XiaoningDing @pdgetrf

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Sep 9, 2018

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Sep 9, 2018

@vinaykul Please use email for questions rather than posting to misc. github issues. For your questions, I suggest reaching out to the autoscaling, apps, and node SIGs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment