Join GitHub today
Cannot force delete busted, empty statefulset #72598
I had applied
I thought I must have made a mistake in a finalizer or something similar, that blocked a pod from shutting down. But I could delete the pods individually, no problem. The STS stayed at
I eventually deleted everything else - all PVCs, PVs, pods, namespaces, services, deployments, and configmaps - and set
I enabled diagnostic logs from the cluster (I'm on Azure), and I can validate that the HTTP DELETE is logged as received and returns a
What you expected to happen:
I expected the STS to be deleted, at least when using
How to reproduce it (as minimally and precisely as possible):
# Create the statefulset. $ kubectl apply -f statefulset-example-with-lb.yml statefulset.apps/nginx created service/nginx created # Install metacontroller. $ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller-rbac.yaml namespace/metacontroller created serviceaccount/metacontroller created clusterrole.rbac.authorization.k8s.io/metacontroller created clusterrolebinding.rbac.authorization.k8s.io/metacontroller created $ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/metacontroller/master/manifests/metacontroller.yaml customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io created customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io created customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io created statefulset.apps/metacontroller created # Create metacontroller hooks. $ kubectl create configmap service-per-pod-hooks -n metacontroller --from-file=hooks configmap/service-per-pod-hooks created # Create service-per-pod service, deployment, decorator controllers from example code. $ kubectl apply -f https://github.com/GoogleCloudPlatform/metacontroller/raw/master/examples/service-per-pod/service-per-pod.yaml decoratorcontroller.metacontroller.k8s.io/service-per-pod created decoratorcontroller.metacontroller.k8s.io/pod-name-label created deployment.apps/service-per-pod created service/service-per-pod created # Try to delete the STS. $ kubectl delete sts nginx statefulset.apps "nginx" deleted # Wait 30 minutes, then: ^C $ kubectl delete sts nginx --force=true --grace-period=0 statefulset.apps "nginx" deleted # Take your dog to the park, then: ^C $ kubectl delete sts nginx --force=true --grace-period=0 -v=7 I0105 01:11:35.146909 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config I0105 01:11:35.148177 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config I0105 01:11:35.149683 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config I0105 01:11:35.153556 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config I0105 01:11:35.154574 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config I0105 01:11:35.155609 20216 loader.go:359] Config loaded from file /home/ohthehugemanatee/.kube/config warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. I0105 01:11:35.155840 20216 round_trippers.go:383] DELETE https://foobar.hcp.westeurope.azmk8s.io:443/apis/apps/v1/namespaces/default/statefulsets/nginx I0105 01:11:35.155847 20216 round_trippers.go:390] Request Headers: I0105 01:11:35.155851 20216 round_trippers.go:393] Authorization: Bearer 1234567891011121314151617181920 I0105 01:11:35.155854 20216 round_trippers.go:393] Content-Type: application/json I0105 01:11:35.155857 20216 round_trippers.go:393] Accept: application/json I0105 01:11:35.155860 20216 round_trippers.go:393] User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946 I0105 01:11:35.631186 20216 round_trippers.go:408] Response Status: 200 OK in 475 milliseconds statefulset.apps "nginx" force deleted I0105 01:11:35.638433 20216 round_trippers.go:383] GET https://foobar.hcp.westeurope.azmk8s.io:443/apis/apps/v1/namespaces/default/statefulsets/nginx I0105 01:11:35.638470 20216 round_trippers.go:390] Request Headers: I0105 01:11:35.638491 20216 round_trippers.go:393] User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946 I0105 01:11:35.638526 20216 round_trippers.go:393] Accept: application/json I0105 01:11:35.638552 20216 round_trippers.go:393] Authorization: Bearer 1234567891011121314151617181920 I0105 01:11:35.746535 20216 round_trippers.go:408] Response Status: 200 OK in 107 milliseconds I0105 01:11:35.771051 20216 round_trippers.go:383] GET https://foobar.hcp.westeurope.azmk8s.io:443/apis/apps/v1/namespaces/default/statefulsets?fieldSelector=metadata.name%3Dnginx&resourceVersion=2557&watch=true I0105 01:11:35.771112 20216 round_trippers.go:390] Request Headers: I0105 01:11:35.771145 20216 round_trippers.go:393] Accept: application/json I0105 01:11:35.771171 20216 round_trippers.go:393] User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946 I0105 01:11:35.771196 20216 round_trippers.go:393] Authorization: Bearer 1234567891011121314151617181920 I0105 01:11:35.856290 20216 round_trippers.go:408] Response Status: 200 OK in 85 milliseconds # Eat dinner, go for a nice walk afterwards, binge watch some Doctor Who, and check back. The command still hasn't returned.
Anything else we need to know?:
I think this is different from #65754 , because
I think this is different from #36333 , because the STS has no pods, and seems to have no trouble detecting that there are no pods left.
I think this is different from #59867, because I don't get any error messages that I can find.
also replicated with client version
@ohthehugemanatee: There are no sig labels on this issue. Please add a sig label by either:
Note: Method 1 will trigger an email to the group. See the group list.
Are there finalizers on the statefulset? (get from the apiserver with
If so, those must be removed by the controller responsible for them before the API object can be deleted.
thank you @liggitt ! that was just the input I needed to find the problem: my DecoratorController had a sync webhook with a typo in the name. When I fixed the typo, everything resolved instantly.
For posterity: You can validate the finalizers on your statefulset with
Feature suggestion: add a state for STS "waiting for finalizers" (or simliar) that can make this kind of hang explicit. This was unnecessarily hard to track down!