-
Notifications
You must be signed in to change notification settings - Fork 321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pv is stuck Terminating due to race condition when csi-attacher removes finalizer and csi-provisioner tries to remove another finalizer #1217
Comments
I appreciate your reminder. I didn't know this issue. Is it easy to reproduce? It may be a blocker to promote this feature to beta. Unfortunately, we have promoted it to beta and enabled it by default. cc @xing-yang @deepakkinni I will investigate this issue next week. Can you provide reproducible steps or a simple test case? |
If we have pre-provisioned PV bound to PVC and gets deleted. PV first move to failed state ( during PVC delete) and then on stuck in Terminating( during PV delete) Below are the provisioner logs:
|
@andyzhangx, there have been changes to use Patch; hopefully, you shouldn't see this error 5.0.1. Can you try it again? |
@deepakkinni we are using patch verb but still seeing above error. |
Could you please post both the attacher and provisioner logs? Post the whole thing instead of just snippets. |
HI @deepakkinni , @shrutinipane has put all the logs here #1235 (comment) |
@Jainbrt ,the attached logs don't show the issue mentioned by the reporter. @andyzhangx, could you try to reproduce this issue on the latest external-provisioner? |
What happened:
pv is stuck Terminating due to race condition when csi-attacher removes finalizer and csi-provisioner tries to remove another finalizer
symptom
pv is stuck Terminating when pvc is deleted with
HonorPVReclaimPolicy
feature gate enabled.process
csi-attacher would remove the finalizer(e.g. external-attacher/disk-csi-azure-com) when pv is detached, and later on csi-provisioner the would try to remove the
external-provisioner.volume.kubernetes.io/finalizer
finalizer when pvc is deleted and since pv object is in the cache of provisioner , the finalizer deletion always fail until maximum 6 retries exceeds, and finally leaves the pv in Terminating state forever (the underlying storage is deleted before finalizer remove fails).remove all finalizers from the pv and then delete pv manually
/kind bug
cc @jsafrane
What you expected to happen:
How to reproduce it:
Anything else we need to know?:
Environment:
kubectl version
): 1.27uname -a
):The text was updated successfully, but these errors were encountered: