Change destroy operation to use foreground cascading delete #2379
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default, Kubernetes uses "background cascading deletion" (BCD) to clean up resources. This works in most cases with the eventual consistency model as resources are garbage collected. However, there are cases where BCD can lead to stuck resources due to race conditions between dependents. A concrete example is an application Deployment that includes a volume mount managed by a Container Storage Interface (CSI) driver. The underlying Pods managed by this Deployment depend on the CSI driver to unmount the volume on teardown, and this process can take some time. Thus, if a Namespace containing both the CSI driver and the application Deployment is deleted, it is possible for the CSI driver to be removed before it has finished tearing down the application Pods, leaving them stuck in a "Terminating" state. A reliable way to avoid this race condition is by using "foreground cascading deletion" (FCD) instead. FCD blocks deletion of the parent resource until any children have been deleted. In the previous example, the application Deployment resource would not be deleted until all of the underlying Pods had unmounted the CSI volume and finished terminating. Once the application Deployment is gone, then Pulumi can safely clean up the CSI driver as well. One downside of this approach is that resource deletion can take longer to resolve since Kubernetes is explicitly waiting on the delete operation to complete. However, this increases reliability of the delete operation by making it less prone to race conditions, so the tradeoff seems worth it.
Does the PR have any schema changes?Looking good! No breaking changes found. |
kpitzen
approved these changes
May 9, 2023
Does the PR have any schema changes?Looking good! No breaking changes found. |
rquitales
approved these changes
May 9, 2023
|
@lblackstone Loved the detailed PR description with motivation and explanation of the change. I'd love to see it extended with the testing approach: why you didn't think we needed extra tests, whether you tested it manually, what kind of risks it could have brought in, etc. |
I added some additional detail to the PR description. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
By default, Kubernetes uses "background cascading deletion" (BCD) to clean up resources. This works in most cases with the eventual consistency model as resources are garbage collected. However, there are cases where BCD can lead to stuck resources due to race conditions between dependents. A concrete example is an application Deployment that includes a volume mount managed by a Container Storage Interface (CSI) driver. The underlying Pods managed by this Deployment depend on the CSI driver to unmount the volume on teardown, and this process can take some time. Thus, if a Namespace containing both the CSI driver and the application Deployment is deleted, it is possible for the CSI driver to be removed before it has finished tearing down the application Pods, leaving them stuck in a "Terminating" state.
A reliable way to avoid this race condition is by using "foreground cascading deletion" (FCD) instead. FCD blocks deletion of the parent resource until any children have been deleted. In the previous example, the application Deployment resource would not be deleted until all of the underlying Pods had unmounted the CSI volume and finished terminating. Once the application Deployment is gone, then Pulumi can safely clean up the CSI driver as well.
One downside of this approach is that resource deletion can take longer to resolve since Kubernetes is explicitly waiting on the delete operation to complete. However, this increases reliability of the delete operation by making it less prone to race conditions, so the tradeoff seems worth it.
This PR doesn't include additional testing because this scenario is already well covered by existing tests. Every test includes a destroy operation, which exercises the new behavior. This change was confirmed to fix the customer issue, and manual testing was also performed.
Related issues (optional)
Fix https://github.com/pulumi/customer-support/issues/931