bug- race condition due to unused resource generation metadata/spec

[muang0]

I would like to propose a new 'ReadinessCheckDelay' argument to the helm commands: 'install', 'upgrade', and 'rollback' based on a race condition we are seeing between helm and a kubernetes controller. We are running helm install with the 'wait' and 'atomic' arguments, but helm doesn't track the resource readiness state (we are seeing this in approximately half of our installs that contain a statefulset). We have reviewed the api-server logs and found that helm makes both the patch and resource readiness get calls before the statefulset controller has the chance to update the resource. This is causing a false positive for the readiness check on the statefulset, and helm incorrectly updates the install status as completed successfully. I am proposing a 'ReadinessCheckDelay' argument that can be used to add some delay between the resources being patched by helm and the subsequent readiness checks. Unfortunately I can't share logs because this is something I am encountering in my cluster at work (highly restrictive fintech). This is something I'm happy to contribute assuming the community is open to this change. Thanks for the feedback :)

Output of helm version: 3.8.0

Output of kubectl version: 1.21.5

Cloud Provider/Platform (AKS, GKE, Minikube etc.): EKS

[joejulian]

If I'm understanding correctly, this affects the upgrade and rollback processes due to the fact that when a StatefulSet or Deployment is updated, it's still considered ready because the subresources (pods, replicasets, etc) have not been updated yet. This fulfills the resources expectations (3 pods ready out of 3, for instance) causing the wait to consider the resource ready.

Have you considered filing a Kubernetes bug for this? It feels like this should be handled in the API. When patching a Deployment, StatefulSet, etc, it seems like maybe it should reset the readiness. I'd be really interested in seeing what sig-api-machinery thinks.

[joejulian]

As for a proposal, that would start as a HIP.

[muang0]

Thanks for the response! Just opened an issue to kubernetes and tagged sig-api-machinery. Let's see what their analysis is and go from there 😄

[muang0]

@joejulian Check out Jordans response on my issue opened to kubernetes. Seems like a great suggestion to incorporate generational checks into the readiness logic. Might be worth a review to see if there are additional resources that could benefit from generational checks. What are your thoughts on this idea and it's potential as a HIP? Thanks for the help again!

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[muang0]

Bumping, this isn't stale