-
Notifications
You must be signed in to change notification settings - Fork 38.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Storage protection feature does not integrate well with StatefulSet PVC recreation #74374
Comments
A workaround in the meantime is to delete the StatefulSet pod a second time if it's stuck in pending due to missing PVC. |
/assign |
/assign msau42 |
Hey @msau42 thank you for having opened the issue and the temporary workaround any news on that bug ? ?? |
I think this need to be fixed, but not just workaround. a second delete of pod is not a graceful operation. |
I would be happy to help guide someone to come up with a possible design and implementation for a fix. /help |
@msau42: Please ensure the request meets the requirements listed here. If this request no longer meets these requirements, the label can be removed In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
when recreate pod, check if the PVC was label with deletion. maybe we could delete the PVC immediately by force and create a new PVC before Pod creation. |
Hey @msau42, I would like to work on this issue. As far as I understand StatefulSetController check the different state of the pods and take some action according to pod states when syncing statefulset. May be I add the pod wdyt? |
the most hard part is that finding the status of pvc(creating pod process), and the deletion of pvc(pvc-protection finalizer) is in different goroutines. there will be concourrent problem. |
Alternative solution might be like https://github.com/mucahitkurt/kubernetes/tree/fix/enable-statefulset-pvc-recreation-for-pending-pod |
I think you mean mucahitkurt@1ab4b97 |
@mucahitkurt if I understand your code well, if a Pod is in |
@mucahitkurt we may need know if the pod keeps in pending or just pending transitorily.
|
@cscetbon I'll test these scenarios on my local cluster, but If the previous PVC is still terminating, I expect that pvc lister returns this pvc and @cwdsuzhou actualy |
I wonder if we actually need to check pod state? Regardless of pod state, can we just check if any of its pvcs are missing? |
@msau42 what if we check that it exists and it does in deleting state ? |
@msau42 , Most probably there won't be any side effect checking the missing PVCs with Checking pod's PVCs existence at every sync operation can create some performance effect. I don't know about the different scenarios that we need to check the existence of pod's PVCs at every statefulset sync. Do you have any scenario? |
An alternative is to get PVC directly from API server instead of cache to avoid the race. The logic would be straightforward and the code change is small. We'll only do this when creating a Pod with an existing PVC. Therefore, the chance of hitting this case would be pretty low, and the impact to API server would be small too. |
@janetkuo |
/sub |
Is this issue fixed in k8s version 1.20? I used to hit this issue 100% of the times in 1.19, in 1.20 pod remains in pending until the pvc gets recreated, without any PVC not found events, once the pvc is recreated pod goes to running. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
/remove-lifecycle rotten |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Is the deletion order of Pod -> PVC -> PV still prone to this bug/race-condition? I still see this behavior in 1.20.5 |
Yes, PVC should be marked deleted before Pod. |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
@cofyc Is this issue completely resolved or still need some fixes after your change? |
/remove-lifecycle rotten |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened:
The storage protection feature prevents PVC deletion until all Pods referencing it are deleted. This doesn't work well with Statefulsets when you want to delete the PVC and have the StatefulSet controller recreate it. The StatefulSet controller only creates missing PVCs when creating a new Pod. So you could have a sequence like this:
Nothing tries to recreate the PVC again and the Pod is stuck in pending forever.
What you expected to happen:
We could potentially add logic in the statefulset controller to check if the PVC is terminating here. However, it uses an informer so the race may still exist.
Another option is to make the StatefulSet controller actively reconcile PVC creation. This would probably require more substantial changes to the controller.
@kubernetes/sig-storage-bugs
@kubernetes/sig-apps-bugs
The text was updated successfully, but these errors were encountered: