helm upgrade --wait does not wait on newer versions

[haljin]

When running helm upgrade --install --wait --timeout 5m0s <release> on a StatefulSet with just one replica helm doesn't wait for pods to redeploy and become ready and just moves on. That causes the automated tests running afterwards to immediately fail as the pod redeploys in the middle of testing.

This seems to happen on version 3.6.3, but works and waits correctly 3.2.4. We can downgrade helm for now, but it'd be nice not to have to micro-manage helm versions on our build agents. :(

[haljin]

Actually, it seems to be happening with all StatefulSets, also ones that have multiple replicas. Maybe because StatefulSets do not have the maxUnavailable parameter?

[mattfarina]

To add some context....

The wait functionality is provided by Kubernetes packages. You can find Helm's use of it at:

helm/pkg/kube/wait.go

Lines 46 to 61 in 2125137

	func (w *waiter) waitForResources(created ResourceList) error {
	w.log("beginning wait for %d resources with timeout of %v", len(created), w.timeout)
	ctx, cancel := context.WithTimeout(context.Background(), w.timeout)
	defer cancel()
	return wait.PollImmediateUntil(2*time.Second, func() (bool, error) {
	for _, v := range created {
	ready, err := w.c.IsReady(ctx, v)
	if !ready || err != nil {
	return false, err
	}
	}
	return true, nil
	}, ctx.Done())
	}

Is the StatefulSet not ready? What version of Helm are you using?

[haljin]

As I mentioned, the operation seems to correctly wait until the pods become ready on 3.2.4 but on 3.6.3 it does not.

[mattfarina]

What version of Kubernetes are you using with this version of Helm? The changes have been in packages provided by the Kubernetes project. I wonder if there is a supported version mismatch.

[haljin]

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"95881afb5df065c250d98cf7f30ee4bb6d281acf", GitTreeState:"clean", BuildDate:"2021-04-25T08:15:25Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.10", GitCommit:"8152330a2b6ca3621196e62966ef761b8f5a61bb", GitTreeState:"clean", BuildDate:"2021-08-11T18:00:37Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

[shwetanshu-cldcvr]

I have a similar issue with Deployments.

Current Behavior:
I have the same issue and it is happening with Deployments. I provided an incorrect container repo to the helm chart, upgrade it using helm and the helm chart gets deployed successfully. However, since I provided incorrect container repo, the the pod fails to start with ErrImagePull. Helm shows upgrade (and even install) as successful. In debug logs, it does not seem to check the status of the pods.

Expected Behavior:
If the container repo is incorrect (or for any reason, pod fails to go into Running status), the Helm Upgrade/Install should fail.

Command Used:
helm upgrade --install test . -f test.yaml --timeout 300s --atomic

Debug Logs:

➜  helm upgrade --install test . -f test.yaml --timeout 300s --wait --atomic --debug

history.go:56: [debug] getting history for release test
upgrade.go:139: [debug] preparing upgrade for test
upgrade.go:147: [debug] performing update for test
upgrade.go:319: [debug] creating upgraded release for test
client.go:218: [debug] checking 7 resources for changes
client.go:501: [debug] Looks like there are no changes for Namespace "service"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "service-ksa"
client.go:501: [debug] Looks like there are no changes for Secret "nservice-docker-registry"
client.go:501: [debug] Looks like there are no changes for ConfigMap "service-configmap"
client.go:501: [debug] Looks like there are no changes for Service "service"
wait.go:48: [debug] beginning wait for 7 resources with timeout of 5m0s
upgrade.go:154: [debug] updating status for upgraded release for test
Release "test" has been upgraded. Happy Helming!

Version:

➜  helm version

version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.17"}
➜  kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.8-gke.2101", GitCommit:"05fc1fdacb292655c77a2058a32ecc6a82cd71b0", GitTreeState:"clean", BuildDate:"2021-08-18T23:13:48Z", GoVersion:"go1.15.13b5", Compiler:"gc", Platform:"linux/amd64"}

[hobbytp]

Any new update here?

[agates4]

Any new update here?

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[muang0]

Potentially related: #10859

[hudzy]

Same here, --wait argument failed to work on some of our CI/CD pipelines, however --atomic argument works fine.

Tested with version 3.8.2, 3.7.2, 3.6.3 and 3.5.4. Oddly, --wait argument is only working correctly on v3.5.4.

[hobbytp]

From the latest info here, the issue is still there for both StatefulSet and Deployment.
Can we re-open this issue?

[psblah]

I'm also seeing this.

Helm version

version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.8"}

Kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11-gke.1100", GitCommit:"20da4c21b3a6b1a56ff6ad5ecb7dee013aaf1b83", GitTreeState:"clean", BuildDate:"2022-04-01T09:40:07Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

[muang0]

For anyone experiencing this issue, check the order of your helm and k8s-controller logs in the api-server. Your symptoms may be due to the race condition described in #10859

If the race condition described in the issue above is the root cause of your 'helm not waiting' behavior, then it will be resolved by #10920