What is the desired behavior of helm regarding manual changes?

[tback]

While dealing with #1844 the question from the subject came up.

Our position in this is best described in the #1844 (comment) by @wjkohnen:

We still can reproduce stale environment variables as mentioned in 1. of #1844 (comment). When we manually add, change, delete (kubectl edit deployment) environment variables, we had expected a chart upgrade to revert any of these changes so as to reflect the state that is expressed in the chart. Any deviance from the expressed state of a chart in our view degrades the idempotency of helm releases and thus the idea of k8s clusters to be "self-healing". After spending some time tackling these issues I have some intuition that this desired behavior is (currently) not feasible. We'd appreciate some clarification.

[technosophos]

Our ideal case is that Helm upgrades would essentially work like Kubernetes apply, but at the release level instead of at the individual resource level. We are gradually getting closer to that (moving) target.

During an upgrade, the chart is rendered into resource definitions. Then a patch set is calculated by comparing those resources to the existing release's resources. Then the patch set is sent to the Kubernetes API server. As with Git, Helm will try to merge the new changes without touching unrelated changes you made with kubectl (in that sense, it "ignores" the changes in-cluster and lets Kubernetes sort it out).

So, for example, if you deploy with replica count 4, then use kubectl to raise the replica count to 6, then use Helm to upgrade the release, the replica count should remain 6.

As a consequence of this, it is not safe to assume that you can use kubectl to make arbitrary changes to a release, and then rely upon Helm to revert those changes. To accomplish that, I believe you'd have to use Helm to delete/rollback. Or, better yet, only use Helm to make changes to your release. Then you will have not only a record of what you changed, but the helm history and helm rollback commands to get you out of a bind.

/cc @adamreese

[cblecker]

What happens if a bad release goes out? I've encountered a situation where a value was typo'd, and Kubernetes rejected the patch set.. but somehow helm thought the change went through, and adjusting it didn't fix the deployment.

[jwkohnen]

@cblecker I have seen something like that as well and I could not reproduce it with helm v2.2.x. See my last comment in #1844.

[cblecker]

@technosophos: RE #2070 (comment) -- is there a way to debug output the patch set to have more insight into what's actually being applied? It's a bit scary for the chart's debug output to desync with what's actually happening in the cluster. A kubectl apply of the whole object was assumed on my part.

[technosophos]

I believe @adamreese is working on that. There's also the helm-diff plugin.

[ahawkins]

We're hitting this with these versions:

workstation $ helm version
Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

We've made kubectl edit fixes in our production environment for:

1. changing Deployment replicas
2. changing Deployment container image tags
3. changing Deployment environment variables
4. changing container resources/limits

Changes were not reverted by the next deploy. This is especially concerning with container image tags. Essentially what that boils down to is that if someone makes a change with kubectl edit, then we cannot be sure that hem upgrade puts things as they are in chart.

@technosophos are you suggesting that we do a rollback to revert those kubectl edit patches before doing the next helm upgrade?

[splisson-altair]

@technosophos I think I understand the logic that Helm is keeping an history of the changes and does not override changes done manually with kubectl but that in fact prevents the use of Helm to establish a known state based on values passed to Helm charts.
Any plan for an option to force helm to apply an upgrade of resources with the values passed ?

[bacongobbler]

Any plan for an option to force helm to apply an upgrade of resources with the values passed ?

You mean like helm upgrade --force?

[splisson-altair]

@bacongobbler --force did not apply the values as I expected it would

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[mark-00]

For my use case this helps
#5202

[woodcockjosh]

I hope helm would not revert changes to the replica count as this would break horizontal autoscalers