Error when updating Statefulsets

[mariusgrigoriu]

The first time I'm updating a stateful set I get an error like Error: UPGRADE FAILED: StatefulSet.apps "eerie-tortoise-consul" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas' are forbidden. on the first try. Subsequent attempts seem to work with no error.

An easy way to repro for us:

1. helm install stable/consul
2. helm upgrade wobbly-bat stable/consul --set=Memory=200Mi
3. Observe error
4. helm upgrade wobbly-bat stable/consul --set=Memory=300Mi
5. No error, statefulset is updated

Client: &version.Version{SemVer:"v2.2.3", GitCommit:"1402a4d6ec9fb349e17b912e32fe259ca21181e3", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.2.3", GitCommit:"1402a4d6ec9fb349e17b912e32fe259ca21181e3", GitTreeState:"clean"}

Logs

2017/03/21 00:16:36 storage.go:133: Getting release history for 'eerie-tortoise'
2017/03/21 00:16:37 release_server.go:913: Executing pre-upgrade hooks for eerie-tortoise
2017/03/21 00:16:37 release_server.go:940: Hooks complete for pre-upgrade eerie-tortoise
2017/03/21 00:16:37 client.go:381: generating strategic merge patch for *runtime.Unstructured
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul-tls
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul-statsdexport-config
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul-config
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul-alerts
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul
2017/03/21 00:16:37 client.go:393: Looks like there are no changes for eerie-tortoise-consul-ing
2017/03/21 00:16:37 client.go:381: generating strategic merge patch for *runtime.Unstructured
2017/03/21 00:16:37 client.go:246: error updating the resource eerie-tortoise-consul:
StatefulSet.apps "eerie-tortoise-consul" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas' are forbidden.
2017/03/21 00:16:37 release_server.go:324: warning: Upgrade "eerie-tortoise" failed: StatefulSet.apps "eerie-tortoise-consul" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas' are forbidden.
2017/03/21 00:16:37 storage.go:53: Updating "eerie-tortoise" (v13) in storage
2017/03/21 00:16:37 storage.go:45: Create release "eerie-tortoise" (v14) in storage
2017/03/21 00:16:45 storage.go:133: Getting release history for 'eerie-tortoise'
2017/03/21 00:16:45 release_server.go:913: Executing pre-upgrade hooks for eerie-tortoise
2017/03/21 00:16:45 release_server.go:940: Hooks complete for pre-upgrade eerie-tortoise
2017/03/21 00:16:45 client.go:381: generating strategic merge patch for *runtime.Unstructured
2017/03/21 00:16:45 client.go:393: Looks like there are no changes for eerie-tortoise-consul-tls
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul-statsdexport-config
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul-config
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul-alerts
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul-ing
2017/03/21 00:16:46 client.go:393: Looks like there are no changes for eerie-tortoise-consul
2017/03/21 00:16:46 release_server.go:913: Executing post-upgrade hooks for eerie-tortoise
2017/03/21 00:16:46 release_server.go:940: Hooks complete for post-upgrade eerie-tortoise
2017/03/21 00:16:46 storage.go:53: Updating "eerie-tortoise" (v14) in storage
2017/03/21 00:16:46 storage.go:45: Create release "eerie-tortoise" (v15) in storage
2017/03/21 00:16:46 storage.go:133: Getting release history for 'eerie-tortoise'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-gossip-key'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-tls'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-statsdexport-config'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-config'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-alerts'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul-ing'
2017/03/21 00:16:47 client.go:155: Doing get for: 'eerie-tortoise-consul'

[technosophos]

When you say "seem to work", do you mean that you have checked with kubectl and can verify that the changes are present in the StatefulSet definition?

[technosophos]

See this explanation in the Kubernetes documentation: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

It states:

As demonstrated in the Scaling a StatefulSet section, the replicas field of a StatefulSet is mutable. The only other field of a StatefulSet that can be updated is the spec.template.containers field.

[mariusgrigoriu]

The changes made were to spec.template.containers.resources so we would expect no errors. We received the error only on the first attempt. Subsequent attempts were verified to stick.

PS. What's the recommended way of updating StatefulSets managed by Helm with changes beyond containers and replicas?

[prat0318]

PS. What's the recommended way of updating StatefulSets managed by Helm with changes beyond containers and replicas?

Also, curious about above ^

[deitch]

Also interested. What is the process when I do need to update a StatefulSet? It cannot be "delete and create", because:

1. It would destroy the entire set, which is not what I am looking for (e.g. want to add a persistent volume, or change memory reservation)
2. It doesn't work for CD pipelines (not going to ask CircleCI or Jenkins or Concourse to parse the kube .yml, look for StatefulSet and delete them before applying!).

[deitch]

Or does 1.7 change the semantics?

[diegows]

Same problem here, but updating the "replicas" value. Statefulset doesn't get updated.

[labeh]

Same Problem here, but with Kubernetes 1.8.x this should be allowed: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

[balboah]

I had a similar error when adding a 2nd container (k8s 1.7.8). Solved it by manually deleting without deleting running pods: kubectl delete sts --cascade=false dev-kafka

then running helm upgrade succeeded and it automatically started rolling up the old pods.

[blakebarnett]

FWIW, still seeing this on k8s 1.8.5 & 1.8.6 with helm 2.7.2 and 2.8.0-rc.1

[cscetbon]

With k8s 1.9 I get the error every time I try to update my chart (incubator/cassandra). The only way I found to update the number of replicas is to use the --reuse-values flag :

$ helm upgrade --reuse-values --set config.cluster_size=6 cassandra incubator/cassandra
Release "cassandra" has been upgraded. Happy Helming!
LAST DEPLOYED: Sun Feb 25 11:29:20 2018
NAMESPACE: cassandra
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                 TYPE       CLUSTER-IP  EXTERNAL-IP  PORT(S)                                       AGE
cassandra-cassandra  ClusterIP  None        <none>       7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP  9m

==> v1beta1/StatefulSet
NAME                 DESIRED  CURRENT  AGE
cassandra-cassandra  6        4        9m
...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[longfei-zhang]

What's the recommended way of updating StatefulSets managed by Helm with changes beyond containers and replicas?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[WintersMichael]

I had this problem with 2.8.2, then upgraded to 2.9.1 and had no issue.

[NilsGriebner]

@MikeSchuette thanks for your answer. I will give it a try :)

[licarth]

I just faced the same issue, with 2.10.0, and it seems the only way I found to fix it is @balboah 's solution (delete sts before helm upgrade).
But I'm still not happy with that, because I don't want my CI/CD pipeline to run that before a failing deployment. I would have to run a rollback to the previous version to recreate the sts, where before, helm would take care of that.
Anyone still facing this ?

[numbsafari]

You may want to review the UpdatePolicy set for your StatefulSet. If it is set to OnDelete, then you will need to use the approach described above. Otherwise, it should be picking up these changes.

[skob]

Still here with image changes

[nickbp]

Seeing inconsistent behavior when upgrading a StatefulSet on Helm 3.2.0.

Changes to env and command were applied successfully, while removal of a Secret from volumes and from volumeMounts was not applied.

As a result the StatefulSet relaunched the pods in a broken state because they referenced a Secret that no longer existed. I was able to clean up the errant volume/volumeMount content that helm had left behind with a kubectl edit statefulset. Is this expected behavior?

[bacongobbler]

Kubernetes only allows a small subset of changes to a StatefulSet once it has been created. We cannot update values that Kubernetes considered immutable. This was pointed out earlier by @technosophos in #2149 (comment). I highly recommend giving that document a read-through.

For those interested in more destructive options (i.e. delete and re-create the resource on an upgrade), please have a look at #7431 and the discussion in #7082.

This is intentional behaviour by Kubernetes, so there's nothing we can do about it. I'm closing this as an intentional design choice by Kubernetes. Thanks!

[nickbp]

As mentioned in the above steps, there was no error, instead the change was silently dropped. This resulted in the cluster falling out of sync with the chart content, which then caused further issues. Additionally this was apparently a legal change as the missing change could be fixed manually with a kubectl edit. I assume helm should have at least presented an error when the change was not applied?

[iAnomaly]

@bacongobbler Hoping you will reopen this. I'm experiencing the same edge case mentioned by @mariusgrigoriu and @nickbp. Specifically, even though Helm v3 fails to upgrade the release because of the reasons you described in #2149 (comment), subsequent updates have no error or failure.

If I attempt the invalid update via kubectl edit for example, it continues to fail on each attempt with the same error.

Would you expect Helm v3 to only fail the first time and never again? My concern here is that maybe Helm is actually marking the release as successful internally in its Secrets state when it was not.

Again, trying to be clear; the issue here is NOT that Helm fails to upgrade an immutable spec field (I understand Helm is limited to Kubernetes' API), the issue is the failure appears to be dropped/swallowed after the first occurrence.

[bacongobbler]

I'll re-open the ticket here for further discussion. I do not have any more information to share other than what's been shared in previous comments in this thread. If you've found a case that may cause a bug, we'd appreciate a fix, or at the very least a test case which others can use to reproduce the issue to determine a fix. Thanks.

[momonala]

I managed to solve this issue by deleting and purging my helm release, then redeploying:

helm delete --purge <RELEASE NAME>

then
helm upgrade --install --wait <CHART>

[ebuildy]

Helm could delete sts automatically without removing pods? (with a flag, such as --force-delete-sts)

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.