-
Notifications
You must be signed in to change notification settings - Fork 7.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Install --wait
should actually wait for extension/v1beta1 Deployments/PODs to be ready
#4210
Comments
@bacongobbler I remember back in Deis (v2 anyway) we did put in a lot more structure and "intelligence" (I'm kinda cringing thinking back at some of the code) to define proper ready (healthcheck, etc) and also pull out errors (pull secrets missing, etc). Can we lift that code and move it into helm? It was working against the API direct, and now with more advances in how Kubernetes ties things together a lot of that logic consolidates anyway. Thoughts? |
Yes, that could be a good solution assuming those assumptions we made back when deployments were in extensions/v1beta1 still run true. Before we go rip out existing code though, let's try to reproduce the error 😄 I found I could reproduce this bug IFF the Deployment apiVersion is How to reproduce this:
We want to set the service type to NodePort so the watcher isn't waiting for the load balancer to become available. If you install this, you can hit this regression. Now, change the apiVersion in traefik/templates/deployment.yaml to
If you run This was tested using v2.9.1. The latest changes to --wait in the last few releases would have been with #3373 or #3921. Either that or it's a regression in the Kubernetes client APIs, which has bitten us in the past. FYI, upgrading to apps/v1 won't work for existing production deployments since Tiller will stick to
For now the best path forward is to upgrade to the |
labelling as a bug/regression. Not sure what our deprecation policy is for older API versions. I'll bring this up in tomorrow's dev call |
--wait
should actually wait for Deployments/PODs to be ready--wait
should actually wait for extension/v1beta1 Deployments/PODs to be ready
also updated the title to reflect the bug that this only affects Deployments using apiVersions prior to apps/v1. |
Oh so the stable version may have finally removed that need! Great. Then it might be more work to fix it up then for people to migrate to stable? |
So, is this only an issue with https://github.com/kubernetes/helm/blob/master/pkg/kube/wait.go#L252 That's why Charts CI additionally uses https://github.com/kubernetes-helm/chart-testing/blob/master/lib/chartlib.sh#L292-L295 |
i did not test apps/v1beta1 but if someone's willing to test that it'd be great! Some of the community charts (traefik like in my example) rely on extensions/v1beta1 so they're gonna hit this bug. |
Seems to do the RightThing and wait using apps/v1beta1.
|
@bacongobbler In terms of supported API versions, the last discussion about the |
It is not waiting for apps/v1 Deployment on Helm 2.14 (client and server). To reproduce just try the script: helm repo add storageos https://charts.storageos.com
helm install storageos/storageos-operator --namespace storageos-operator --wait
kubectl patch storageclass fast -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' It installs a apiVersion: apps/v1 Deployment. If you wait some seconds you can patch the created StorageClass. |
A new command "system application-update" is introduced in this commit to support updating an applied application to a new version with a new versioned app tarfile. The application update leverages the existing application upload workflow to first validating/uploading the new app tarfile, then invokes Armada apply or rollback to deploy the charts for the new versioned application. If the version has ever applied before, Armada rollback will be performed, otherwise, Armada apply will be performed. After apply/rollback to the new version is done, the files for the old application version will be cleaned up as well as the releases which are not in the new application version. Once the update is completed successfully, the status will be set to "applied" so that user can continue applying app with user overrides. If there has any failure during updating, application recover will be triggered to recover the app to the old version. If application recover fails, the application status will be populated to "apply-failed" so that user can re-apply app. In order to use Armada rollback, a new sysinv table "kube_app_releases" is created to record deployed helm releases versions. After each app apply, if any helm release version changed, the corresponding release needs to be updated in sysinv db as well. The application overrides have been changed to tie to a specific application in commit https://review.opendev.org/#/c/660498/. Therefore, the user overrides is preserved when updating. Note: On the AIO-SX, always use Armada apply even it was applied issue on AIO-SX(replicas is 1) to leverage rollback, Armada/helm rollback --wait does not wait for pods to be ready before it returns. Related helm issue, helm/helm#4210 helm/helm#2006 Tests conducted(AIO-SX, DX, Standard): - functional tests (both stx-openstack and simple custom app) - upload stx-openstack-1.0-13-centos-stable-latest tarfile which uses latest docker images - apply stx-openstack - update to stx-openstack-1.0-13-centos-stable-versioned which uses versioned docker images - update back to stx-openstack-1.0-13-centos-stable-latest - update to a version that has less/more charts compared to the old version - remove stx-openstack - delete stx-openstack - failure tests - application-update rejected (app not found, update to a same version, operation not permitted etc...) - application-update fails that trigger recover - upload failure ie. invalid tarfile, manifest file validation failed ... - apply/rollback failure ie. download images failure, Armada apply/rollback fails Change-Id: I4e094427e673639e2bdafd8c476b897b7b4327a3 Story: 2005350 Task: 33568 Signed-off-by: Angie Wang <angie.wang@windriver.com>
A new command "system application-update" is introduced in this commit to support updating an applied application to a new version with a new versioned app tarfile. The application update leverages the existing application upload workflow to first validating/uploading the new app tarfile, then invokes Armada apply or rollback to deploy the charts for the new versioned application. If the version has ever applied before, Armada rollback will be performed, otherwise, Armada apply will be performed. After apply/rollback to the new version is done, the files for the old application version will be cleaned up as well as the releases which are not in the new application version. Once the update is completed successfully, the status will be set to "applied" so that user can continue applying app with user overrides. If there has any failure during updating, application recover will be triggered to recover the app to the old version. If application recover fails, the application status will be populated to "apply-failed" so that user can re-apply app. In order to use Armada rollback, a new sysinv table "kube_app_releases" is created to record deployed helm releases versions. After each app apply, if any helm release version changed, the corresponding release needs to be updated in sysinv db as well. The application overrides have been changed to tie to a specific application in commit https://review.opendev.org/#/c/660498/. Therefore, the user overrides is preserved when updating. Note: On the AIO-SX, always use Armada apply even it was applied issue on AIO-SX(replicas is 1) to leverage rollback, Armada/helm rollback --wait does not wait for pods to be ready before it returns. Related helm issue, helm/helm#4210 helm/helm#2006 Tests conducted(AIO-SX, DX, Standard): - functional tests (both stx-openstack and simple custom app) - upload stx-openstack-1.0-13-centos-stable-latest tarfile which uses latest docker images - apply stx-openstack - update to stx-openstack-1.0-13-centos-stable-versioned which uses versioned docker images - update back to stx-openstack-1.0-13-centos-stable-latest - update to a version that has less/more charts compared to the old version - remove stx-openstack - delete stx-openstack - failure tests - application-update rejected (app not found, update to a same version, operation not permitted etc...) - application-update fails that trigger recover - upload failure ie. invalid tarfile, manifest file validation failed ... - apply/rollback failure ie. download images failure, Armada apply/rollback fails Change-Id: I4e094427e673639e2bdafd8c476b897b7b4327a3 Story: 2005350 Task: 33568 Signed-off-by: Angie Wang <angie.wang@windriver.com>
A new command "system application-update" is introduced in this commit to support updating an applied application to a new version with a new versioned app tarfile. The application update leverages the existing application upload workflow to first validating/uploading the new app tarfile, then invokes Armada apply or rollback to deploy the charts for the new versioned application. If the version has ever applied before, Armada rollback will be performed, otherwise, Armada apply will be performed. After apply/rollback to the new version is done, the files for the old application version will be cleaned up as well as the releases which are not in the new application version. Once the update is completed successfully, the status will be set to "applied" so that user can continue applying app with user overrides. If there has any failure during updating, application recover will be triggered to recover the app to the old version. If application recover fails, the application status will be populated to "apply-failed" so that user can re-apply app. In order to use Armada rollback, a new sysinv table "kube_app_releases" is created to record deployed helm releases versions. After each app apply, if any helm release version changed, the corresponding release needs to be updated in sysinv db as well. The application overrides have been changed to tie to a specific application in commit https://review.opendev.org/#/c/660498/. Therefore, the user overrides is preserved when updating. Note: On the AIO-SX, always use Armada apply even it was applied issue on AIO-SX(replicas is 1) to leverage rollback, Armada/helm rollback --wait does not wait for pods to be ready before it returns. Related helm issue, helm/helm#4210 helm/helm#2006 Tests conducted(AIO-SX, DX, Standard): - functional tests (both stx-openstack and simple custom app) - upload stx-openstack-1.0-13-centos-stable-latest tarfile which uses latest docker images - apply stx-openstack - update to stx-openstack-1.0-13-centos-stable-versioned which uses versioned docker images - update back to stx-openstack-1.0-13-centos-stable-latest - update to a version that has less/more charts compared to the old version - remove stx-openstack - delete stx-openstack - failure tests - application-update rejected (app not found, update to a same version, operation not permitted etc...) - application-update fails that trigger recover - upload failure ie. invalid tarfile, manifest file validation failed ... - apply/rollback failure ie. download images failure, Armada apply/rollback fails Change-Id: I4e094427e673639e2bdafd8c476b897b7b4327a3 Story: 2005350 Task: 33568 Signed-off-by: Angie Wang <angie.wang@windriver.com>
closing as inactive. |
The helm CLI has
--wait
argument for install:As can be seen with testing, it only "waits" on
Jobs
.PODs
are not waited on to be "in a ready state" per the CLI docs. Helm is acting like a "fire and forget" -- it will not wait for the Deployment created PODs to become ready (per their readiness probes).The published documentation seems to contradict the helm CLI usage-help, and does accurately document the seen behavior: https://docs.helm.sh/developing_charts/
What does it mean to wait until a hook is ready? This depends on the resource declared in the hook. If the resources is a Job kind, Tiller will wait until the job successfully runs to completion. And if the job fails, the release will fail. This is a blocking operation, so the Helm client will pause while the Job is run.
For all other kinds, as soon as Kubernetes marks the resource as loaded (added or updated), the resource is considered “Ready”.
Output from my
install --wait
:Note: The output clearly shows my
cb-core-auth
deployment POD as not ready (0/1
). The behavior appear to only wait for the Jobs to complete.Output of
helm version
:Output of
kubectl version
:Cloud Provider/Platform (AKS, GKE, Minikube etc.):
docker-edge (on MacOS)
The text was updated successfully, but these errors were encountered: