Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is the desired behavior of helm regarding manual changes? #2070

Closed
tback opened this issue Mar 6, 2017 · 14 comments
Closed

What is the desired behavior of helm regarding manual changes? #2070

tback opened this issue Mar 6, 2017 · 14 comments

Comments

@tback
Copy link
Contributor

tback commented Mar 6, 2017

While dealing with #1844 the question from the subject came up.

Our position in this is best described in the #1844 (comment) by @wjkohnen:

We still can reproduce stale environment variables as mentioned in 1. of #1844 (comment). When we manually add, change, delete (kubectl edit deployment) environment variables, we had expected a chart upgrade to revert any of these changes so as to reflect the state that is expressed in the chart. Any deviance from the expressed state of a chart in our view degrades the idempotency of helm releases and thus the idea of k8s clusters to be "self-healing". After spending some time tackling these issues I have some intuition that this desired behavior is (currently) not feasible. We'd appreciate some clarification.

@tback tback changed the title What is the desired behavior of helm regarding the presence of manual changes? What is the desired behavior of helm regarding manual changes? Mar 7, 2017
@technosophos
Copy link
Member

technosophos commented Mar 7, 2017

Our ideal case is that Helm upgrades would essentially work like Kubernetes apply, but at the release level instead of at the individual resource level. We are gradually getting closer to that (moving) target.

During an upgrade, the chart is rendered into resource definitions. Then a patch set is calculated by comparing those resources to the existing release's resources. Then the patch set is sent to the Kubernetes API server. As with Git, Helm will try to merge the new changes without touching unrelated changes you made with kubectl (in that sense, it "ignores" the changes in-cluster and lets Kubernetes sort it out).

So, for example, if you deploy with replica count 4, then use kubectl to raise the replica count to 6, then use Helm to upgrade the release, the replica count should remain 6.

As a consequence of this, it is not safe to assume that you can use kubectl to make arbitrary changes to a release, and then rely upon Helm to revert those changes. To accomplish that, I believe you'd have to use Helm to delete/rollback. Or, better yet, only use Helm to make changes to your release. Then you will have not only a record of what you changed, but the helm history and helm rollback commands to get you out of a bind.

/cc @adamreese

@cblecker
Copy link
Contributor

cblecker commented Mar 10, 2017

What happens if a bad release goes out? I've encountered a situation where a value was typo'd, and Kubernetes rejected the patch set.. but somehow helm thought the change went through, and adjusting it didn't fix the deployment.

@jwkohnen
Copy link

jwkohnen commented Mar 14, 2017

@cblecker I have seen something like that as well and I could not reproduce it with helm v2.2.x. See my last comment in #1844.

@cblecker
Copy link
Contributor

cblecker commented Mar 14, 2017

@technosophos: RE #2070 (comment) -- is there a way to debug output the patch set to have more insight into what's actually being applied? It's a bit scary for the chart's debug output to desync with what's actually happening in the cluster. A kubectl apply of the whole object was assumed on my part.

@technosophos
Copy link
Member

technosophos commented Mar 15, 2017

I believe @adamreese is working on that. There's also the helm-diff plugin.

@ahawkins
Copy link
Contributor

ahawkins commented Jul 19, 2017

We're hitting this with these versions:

workstation $ helm version
Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

We've made kubectl edit fixes in our production environment for:

  1. changing Deployment replicas
  2. changing Deployment container image tags
  3. changing Deployment environment variables
  4. changing container resources/limits

Changes were not reverted by the next deploy. This is especially concerning with container image tags. Essentially what that boils down to is that if someone makes a change with kubectl edit, then we cannot be sure that hem upgrade puts things as they are in chart.

@technosophos are you suggesting that we do a rollback to revert those kubectl edit patches before doing the next helm upgrade?

@splisson-altair
Copy link

splisson-altair commented Sep 20, 2017

@technosophos I think I understand the logic that Helm is keeping an history of the changes and does not override changes done manually with kubectl but that in fact prevents the use of Helm to establish a known state based on values passed to Helm charts.
Any plan for an option to force helm to apply an upgrade of resources with the values passed ?

@bacongobbler
Copy link
Member

bacongobbler commented Sep 20, 2017

Any plan for an option to force helm to apply an upgrade of resources with the values passed ?

You mean like helm upgrade --force?

@splisson-altair
Copy link

splisson-altair commented Sep 20, 2017

@bacongobbler --force did not apply the values as I expected it would

@fejta-bot
Copy link

fejta-bot commented Jan 6, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot
Copy link

fejta-bot commented Feb 9, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot
Copy link

fejta-bot commented Mar 11, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@mark-00
Copy link

mark-00 commented Jan 24, 2019

For my use case this helps
#5202

@woodcockjosh
Copy link

woodcockjosh commented Oct 27, 2019

I hope helm would not revert changes to the replica count as this would break horizontal autoscalers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests