New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deleting namespace stuck at "Terminating" state #60807

Open
shean-guangchang opened this Issue Mar 5, 2018 · 59 comments

Comments

Projects
None yet
@shean-guangchang
Copy link

shean-guangchang commented Mar 5, 2018

I am using v1.8.4 and I am having the problem that deleted namespace stays at "Terminating" state forever. I did "kubectl delete namespace XXXX" already.

@dims

This comment has been minimized.

Copy link
Member

dims commented Mar 7, 2018

/sig api-machinery

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Mar 10, 2018

@shean-guangchang Do you have some way to reproduce this?

And out of curiosity, are you using any CRDs? We faced this problem with TPRs previously.

@nikhita

This comment has been minimized.

Copy link
Member

nikhita commented Mar 10, 2018

/kind bug

@justinbarrick

This comment has been minimized.

Copy link

justinbarrick commented Mar 14, 2018

I seem to be experiencing this issue with a rook deployment:

➜  tmp git:(master) ✗ kubectl delete namespace rook
Error from server (Conflict): Operation cannot be fulfilled on namespaces "rook": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
➜  tmp git:(master) ✗ 

I think it does have something to do with their CRD, I see this in the API server logs:

E0314 07:28:18.284942       1 crd_finalizer.go:275] clusters.rook.io failed with: timed out waiting for the condition
E0314 07:28:18.287629       1 crd_finalizer.go:275] clusters.rook.io failed with: Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io "clusters.rook.io": the object has been modified; please apply your changes to the latest version and try again

I've deployed rook to a different namespace now, but I'm not able to create the cluster CRD:

➜  tmp git:(master) ✗ cat rook/cluster.yaml 
apiVersion: rook.io/v1alpha1
kind: Cluster
metadata:
  name: rook
  namespace: rook-cluster
spec:
  dataDirHostPath: /var/lib/rook-cluster-store
➜  tmp git:(master) ✗ kubectl create -f rook/
Error from server (MethodNotAllowed): error when creating "rook/cluster.yaml": the server does not allow this method on the requested resource (post clusters.rook.io)

Seems like the CRD was never cleaned up:

➜  tmp git:(master) ✗ kubectl get customresourcedefinitions clusters.rook.io -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: 2018-02-28T06:27:45Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-03-14T07:36:10Z
  finalizers:
  - customresourcecleanup.apiextensions.k8s.io
  generation: 1
  name: clusters.rook.io
  resourceVersion: "9581429"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/clusters.rook.io
  uid: 7cd16376-1c50-11e8-b33e-aeba0276a0ce
spec:
  group: rook.io
  names:
    kind: Cluster
    listKind: ClusterList
    plural: clusters
    singular: cluster
  scope: Namespaced
  version: v1alpha1
status:
  acceptedNames:
    kind: Cluster
    listKind: ClusterList
    plural: clusters
    singular: cluster
  conditions:
  - lastTransitionTime: 2018-02-28T06:27:45Z
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: 2018-02-28T06:27:45Z
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  - lastTransitionTime: 2018-03-14T07:18:18Z
    message: CustomResource deletion is in progress
    reason: InstanceDeletionInProgress
    status: "True"
    type: Terminating
➜  tmp git:(master) ✗ 
@justinbarrick

This comment has been minimized.

Copy link

justinbarrick commented Mar 14, 2018

I have a fission namespace in a similar state:

➜  tmp git:(master) ✗ kubectl delete namespace fission
Error from server (Conflict): Operation cannot be fulfilled on namespaces "fission": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
➜  tmp git:(master) ✗ kubectl get pods -n fission     
NAME                          READY     STATUS        RESTARTS   AGE
storagesvc-7c5f67d6bd-72jcf   0/1       Terminating   0          8d
➜  tmp git:(master) ✗ kubectl delete pod/storagesvc-7c5f67d6bd-72jcf --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (NotFound): pods "storagesvc-7c5f67d6bd-72jcf" not found
➜  tmp git:(master) ✗ kubectl describe pod -n fission storagesvc-7c5f67d6bd-72jcf
Name:                      storagesvc-7c5f67d6bd-72jcf
Namespace:                 fission
Node:                      10.13.37.5/10.13.37.5
Start Time:                Tue, 06 Mar 2018 07:03:06 +0000
Labels:                    pod-template-hash=3719238268
                           svc=storagesvc
Annotations:               <none>
Status:                    Terminating (expires Wed, 14 Mar 2018 06:41:32 +0000)
Termination Grace Period:  30s
IP:                        10.244.2.240
Controlled By:             ReplicaSet/storagesvc-7c5f67d6bd
Containers:
  storagesvc:
    Container ID:  docker://3a1350f6e4871b1ced5c0e890e37087fc72ed2bc8410d60f9e9c26d06a40c457
    Image:         fission/fission-bundle:0.4.1
    Image ID:      docker-pullable://fission/fission-bundle@sha256:235cbcf2a98627cac9b0d0aae6e4ea4aac7b6e6a59d3d77aaaf812eacf9ef253
    Port:          <none>
    Command:
      /fission-bundle
    Args:
      --storageServicePort
      8000
      --filePath
      /fission
    State:          Terminated
      Exit Code:    0
      Started:      Mon, 01 Jan 0001 00:00:00 +0000
      Finished:     Mon, 01 Jan 0001 00:00:00 +0000
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /fission from fission-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from fission-svc-token-zmsxx (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  fission-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  fission-storage-pvc
    ReadOnly:   false
  fission-svc-token-zmsxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  fission-svc-token-zmsxx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
➜  tmp git:(master) ✗ 

Fission also uses CRDs, however, they appear to be cleaned up.

@barakAtSoluto

This comment has been minimized.

Copy link

barakAtSoluto commented Mar 22, 2018

@shean-guangchang - I had the same issue. I've deleted everything under the namespaces manually, deleted and purged everything from "helm" and restarted the master nodes one by one and it fixed the issue.

I imagine what i've encountered has something to do with "ark", "tiller" and Kuberenets all working together (i bootstraped using helm and backed-up using ark) so this may not be a Kuberenets issue per say, on the other hand, it was pretty much impossible to troubleshot because there are no relevant logs.

@xetys

This comment has been minimized.

Copy link

xetys commented Mar 23, 2018

if it is the rook one, take a look at this: rook/rook#1488 (comment)

@justinbarrick

This comment has been minimized.

Copy link

justinbarrick commented Mar 23, 2018

I guess that makes sense, but it seems buggy that it's possible to get a namespace into an undeletable state.

@OguzPastirmaci

This comment has been minimized.

Copy link
Contributor

OguzPastirmaci commented Apr 26, 2018

I have a similar environment (Ark & Helm) with @barakAtSoluto and have the same issue. Purging and restarting the masters didn't fix it for me though. Still stuck at terminating.

@barakAtSoluto

This comment has been minimized.

Copy link

barakAtSoluto commented Apr 29, 2018

I had that too when trying to recreate the problem. I eventually had to create a new cluster....
Exclude - default, kube-system/public and all ark related namespaces from backup and restore to prevent this from happening...

@jaxxstorm

This comment has been minimized.

Copy link
Contributor

jaxxstorm commented May 3, 2018

I'm also seeing this too, on a cluster upgraded from 1.8.4 to 1.9.6. I don't even know what logs to look at

@adampl

This comment has been minimized.

Copy link

adampl commented Jun 27, 2018

The same issue on 1.10.1 :(

@siXor

This comment has been minimized.

Copy link

siXor commented Jun 28, 2018

Same issue on 1.9.6

Edit: The namespace couldn't be deleted because of some pods hanging. I did a delete with --grace-period=0 --force on them all and after a couple of minutes the namespace was deleted as well.

@xetys

This comment has been minimized.

Copy link

xetys commented Jun 28, 2018

Hey,

I've got this over and over again and it's most of the time some trouble with finalizers.

If a namespace is stuck, try to kubectl get namespace XXX -o yaml and check if there is a finalizer on it. If so, edit the namespace and remove the finalizer (by passing an empty array) and then the namespace gets deleted

@adampl

This comment has been minimized.

Copy link

adampl commented Jun 29, 2018

@xetys is it safe? in my case there is only one finalizer named "kubernetes".

@xetys

This comment has been minimized.

Copy link

xetys commented Jul 2, 2018

That's strange, I've never seen such a finalizer. I just can speak based in my experience. I did that several time in a production cluster and it's still alive

@andraxylia

This comment has been minimized.

Copy link

andraxylia commented Jul 3, 2018

Same issue on 1.10.5. I tried all advice in this issue without result. I was able to get rid of the pods, but the namespace is still hanging.

@andraxylia

This comment has been minimized.

Copy link

andraxylia commented Jul 3, 2018

Actually, the ns too got deleted after a while.

@andraxylia

This comment has been minimized.

Copy link

andraxylia commented Jul 3, 2018

It would be good to understand what causes this behavior, the only finalizer I had is kubernetes. I also have dynamic webhooks, can these be related?

@adampl

This comment has been minimized.

Copy link

adampl commented Jul 3, 2018

@xetys Well, finally I used your trick on the replicas inside that namespace. They had some custom finalizer that probably no longer existed, so I couldn't delete them. When I removed the references to that finalizer, they disappeared and so did the namespace. Thanks! :)

@bobhenkel

This comment has been minimized.

Copy link

bobhenkel commented Jul 13, 2018

Same issue on an EKS 1.10.3 cluster:

Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-28T20:13:43Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
@ManifoldFR

This comment has been minimized.

Copy link

ManifoldFR commented Jul 22, 2018

Having the same problem on a bare metal cluster:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

My namespace looks like so:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"creneaux-app","namespace":""}}
  name: creneaux-app
spec:
  finalizers:
  - kubernetes

It's actually the second namespace I've had have this problem.

@adampl

This comment has been minimized.

Copy link

adampl commented Jul 23, 2018

Try this to get the actual list of all things in your namespace: kubernetes/kubectl#151 (comment)

Then for each object do kubectl delete or kubectl edit to remove finalizers.

@pauloeliasjr

This comment has been minimized.

Copy link

pauloeliasjr commented Jul 24, 2018

removing the initializer did the trick for me...

@ManifoldFR

This comment has been minimized.

Copy link

ManifoldFR commented Jul 24, 2018

When I do kubectl edit namespace annoying-namespace-to-delete and remove the finalizers, they get re-added when I check with a kubectl get -o yaml.

Also, when trying what you suggested @adampl I get no output (removing --ignore-not-found confirms no resources are found in the namespace, of any type).

@slassh

This comment has been minimized.

Copy link

slassh commented Jul 28, 2018

@ManifoldFR , I had the same issue as yours and I managed to make it work by making an API call with json file .
kubectl get namespace annoying-namespace-to-delete -o json > tmp.json
then edit tmp.json and remove"kubernetes"

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

and it should delete your namespace,

@thsheep

This comment has been minimized.

Copy link

thsheep commented Aug 29, 2018

@slassh Perfect solution! thank you very much!!!!

@palmerabollo

This comment has been minimized.

Copy link

palmerabollo commented Oct 12, 2018

I also see this issue with 1.10.x. I find @slassh's comment a workaround that only hides the real issue. Why are the namespaces stuck at Terminating?

@javierprovecho

This comment has been minimized.

Copy link

javierprovecho commented Oct 15, 2018

We discovered the reason for deleting namespaces to be stuck in our case (@palmerabollo)

When a namespace has a finalizer kubernetes, it means its an internal problem with the API Server.

Run kubectl api-resources, if it returns an like the following, it means that the custom API isn't reachable.

error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Run kubectl get apiservices v1beta1.metrics.k8s.io -o yaml, for checking its status conditions.

status:
  conditions:
  - lastTransitionTime: 2018-10-15T08:14:15Z
    message: endpoints for service/metrics-server in "kube-system" have no addresses
    reason: MissingEndpoints
    status: "False"
    type: Available

The above error should be caused by a crashloopbackoff affecting metrics-server. It would be similar for other custom APIs registered in Kubernetes.

Check your services health in kube-system for restoring cluster runtime operations like deleting namespaces.

@grendach

This comment has been minimized.

Copy link

grendach commented Nov 2, 2018

I'm facing with this issue on v1.11.3. At to the finalizers only kubernetes present for problematic namespace.

spec:
  finalizers:
  - kubernetes
@davidedal

This comment has been minimized.

Copy link

davidedal commented Nov 8, 2018

@slassh Thanks a million, your solution works well!
I have the same problem in my cluster with ark, tiller and kubed. I suspect the issues might be the api of kubed that is giving an error, although not sure why it impacts the deletion of another namespace.

@2rs2ts

This comment has been minimized.

Copy link
Contributor

2rs2ts commented Nov 15, 2018

@javierprovecho I was merely playing around with metrics server and since it wasn't working I tried to delete the service and whatnot but my namespace still won't delete, error is

status:
  conditions:
  - lastTransitionTime: 2018-08-24T08:59:36Z
    message: service/metrics-server in "kube-system" is not present
    reason: ServiceNotFound
    status: "False"
    type: Available

Do you know how to recover from this state?

edit: I found out... I had to delete everything even remotely related to metrics/HPA and then restart the entire control plane (had to take down all my replicas of it, before booting them back up.) this included deleting the apiservice v1beta1.metrics.k8s.io itself.

@antoineco

This comment has been minimized.

Copy link
Contributor

antoineco commented Nov 15, 2018

@2rs2ts

$ kubectl delete apiservice v1beta1.metrics.k8s.io

By getting rid of the non-functioning metrics API service the controller-manager will be able to delete the stale namespace(s).

Restarting the control plane is not necessary.

@2rs2ts

This comment has been minimized.

Copy link
Contributor

2rs2ts commented Nov 15, 2018

@antoineco no, it was necessary; I deleted the apiservice and waited quite a while but the namespace would not be deleted until I restarted the control plane.

@shdowofdeath

This comment has been minimized.

Copy link

shdowofdeath commented Nov 19, 2018

first, take small coffee and relax , now go to your k8s master nodes

  1. kubectl cluster-info
    Kubernetes master is running at https://localhost:6443
    KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  1. now run the kube-proxy
    kubectl proxy &
    Starting to serve on 127.0.0.1:8001

save the ID to delete it later on :)
3. find your name-space that decided no to be deleted :) for us it will be cattle-system
kubectl get ns
cattle-system Terminating 1d

put it in file
4. kubectl get namespace cattle-system -o json > tmp.json

  1. edit the file and remove the finalizers
    },
    "spec": {
    "finalizers": [
    "kubernetes"
    ]
    },
    after editing it should look like this 👍
    },
    "spec": {
    "finalizers": [
    ]
    },
    we almost there 👍

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${NAMESPACE}/finalize

and it's gone 👍

@javierprovecho

This comment has been minimized.

Copy link

javierprovecho commented Nov 19, 2018

Hey, the finalizer kubernetes is there for a reason. For us it was a wrongly configured metrics API service name. Maybe for you is something else, which you can discover by looking at your control plane logs. Without confirmation of a bug, removing the finalizer may produce undesired consequences like leaving stuff created that can no longer be accesible again for deletion purposes.

@scones

This comment has been minimized.

Copy link

scones commented Nov 20, 2018

as this issue is still open:
within my minikube cluster running with "none", this happened after the host woke up from hibernate.

my assumption:
in my case the hibernate triggered the same problems, an enabled swap would do.

which yields the assumption:
the swap might be enabled in the affected clusters?

but this is just conjecture. the important thing for me, and anyone landing in this bug with my local setup: hibernate is bad for kubernetes.

@omerElezra

This comment has been minimized.

Copy link

omerElezra commented Nov 20, 2018

first, take small coffee and relax , now go to your k8s master nodes

  1. kubectl cluster-info
    Kubernetes master is running at https://localhost:6443
    KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  1. now run the kube-proxy
    kubectl proxy &
    Starting to serve on 127.0.0.1:8001

save the ID to delete it later on :)
3. find your name-space that decided no to be deleted :) for us it will be cattle-system
kubectl get ns
cattle-system Terminating 1d

put it in file
4. kubectl get namespace cattle-system -o json > tmp.json

  1. edit the file and remove the finalizers
    },
    "spec": {
    "finalizers": [
    "kubernetes"
    ]
    },
    after editing it should look like this 👍
    },
    "spec": {
    "finalizers": [
    ]
    },
    we almost there 👍

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${NAMESPACE}/finalize

and it's gone 👍

Great!!
Works

@wolfadactyl

This comment has been minimized.

Copy link

wolfadactyl commented Nov 20, 2018

I run into this issue periodically if we change our gcloud instances (e.g. upgrading nodes). This replaces the old node from gcloud instances list with a new one but leaves the pods in the k8s namespace hanging:

Reason:                    NodeLost
Message:                   Node <old node> which was running pod <pod> is unresponsive

This then leaves the pods in an unknown state:

$ kubectl get po
NAME                               READY     STATUS    RESTARTS   AGE
<pod>                              2/2       Unknown   0          39d

Due to this, the namespace will never finish terminating. Not sure if this means we should change our finalizers or if there's an actual bug related to terminating that should be handling pods in an UNKNOWN state (or if there should be a way of force terminating a namespace for cases like this).

@shdowofdeath

This comment has been minimized.

Copy link

shdowofdeath commented Nov 21, 2018

I run into this issue periodically if we change our gcloud instances (e.g. upgrading nodes). This replaces the old node from gcloud instances list with a new one but leaves the pods in the k8s namespace hanging:

Reason:                    NodeLost
Message:                   Node <old node> which was running pod <pod> is unresponsive

This then leaves the pods in an unknown state:

$ kubectl get po
NAME                               READY     STATUS    RESTARTS   AGE
<pod>                              2/2       Unknown   0          39d

Due to this, the namespace will never finish terminating. Not sure if this means we should change our finalizers or if there's an actual bug related to terminating that should be handling pods in an UNKNOWN state (or if there should be a way of force terminating a namespace for cases like this).

cool it's not the same issue
you need to put nodes in maintenance mode and then after it's in maintenance mode all pods will be evacuated and then u can delete/upgrade

@sillky

This comment has been minimized.

Copy link

sillky commented Nov 24, 2018

look it ,https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/,
edit resource and delete metadata.finalizers,and delete unuseful crd,you can delete it force

@samuela

This comment has been minimized.

Copy link

samuela commented Dec 9, 2018

But what does the kubernetes finalizer do exactly? Is there any risk that resources are not being correctly cleaned up with this hack?

@marcstreeter

This comment has been minimized.

Copy link

marcstreeter commented Dec 13, 2018

For the rook stuck terminating this helped https://github.com/rook/rook/blob/master/Documentation/ceph-teardown.md

@LeiYangGH

This comment has been minimized.

Copy link

LeiYangGH commented Dec 18, 2018

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

Error from server (NotFound): namespaces "annoying-namespace-to-delete" not found

@LeiYangGH

This comment has been minimized.

Copy link

LeiYangGH commented Dec 18, 2018

first, take small coffee and relax , now go to your k8s master nodes

  1. kubectl cluster-info
    Kubernetes master is running at https://localhost:6443
    KubeDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  1. now run the kube-proxy
    kubectl proxy &
    Starting to serve on 127.0.0.1:8001

save the ID to delete it later on :)
3. find your name-space that decided no to be deleted :) for us it will be cattle-system
kubectl get ns
cattle-system Terminating 1d

put it in file
4. kubectl get namespace cattle-system -o json > tmp.json

  1. edit the file and remove the finalizers
    },
    "spec": {
    "finalizers": [
    "kubernetes"
    ]
    },
    after editing it should look like this 👍
    },
    "spec": {
    "finalizers": [
    ]
    },
    we almost there 👍

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${NAMESPACE}/finalize

and it's gone 👍

Invalid value: "The edited file failed validation": ValidationError(Namespace.spec): invalid type for io.k8s.api.core.v1.NamespaceSpec: got "string", expected "map"

@olmoser

This comment has been minimized.

Copy link

olmoser commented Dec 19, 2018

If you have many namespaces stuck in Terminating, you can automate this:

kubectl get ns | grep Terminating | awk '{print $1}' | gxargs  -n1 -- bash -c 'kubectl get ns "$0" -o json | jq "del(.spec.finalizers[0])" > "$0.json"; curl -k -H "Content-Type: application/json" -X PUT --data-binary @"$0.json" "http://127.0.0.1:8001/api/v1/namespaces/$0/finalize" '

make sure that all namespaces that you want the finalizer removed are indeed Terminating.

You need the kubectl proxy running and jq for the above to work.

@manojbadam

This comment has been minimized.

Copy link

manojbadam commented Jan 16, 2019

In our case, metrics api service is down and i can see this error log from verbose logging

kubectl delete ns <namespace-name> -v=7
.......
I0115 11:03:25.548299   12445 round_trippers.go:383] GET https://<api-server-url>/apis/metrics.k8s.io/v1beta1?timeout=32s
I0115 11:03:25.548317   12445 round_trippers.go:390] Request Headers:
I0115 11:03:25.548323   12445 round_trippers.go:393]     Accept: application/json, */*
I0115 11:03:25.548329   12445 round_trippers.go:393]     User-Agent: kubectl/v1.11.3 (darwin/amd64) kubernetes/a452946
I0115 11:03:25.580116   12445 round_trippers.go:408] Response Status: 503 Service Unavailable in 31 milliseconds

After fixing the metrics apiservice, terminating ones are completed.
Not really sure why deletion depends on metrics apiservice, also intrested to know how it works if metrics apiservice is not installed on the cluster

@javierprovecho

This comment has been minimized.

Copy link

javierprovecho commented Jan 17, 2019

Not really sure why deletion depends on metrics apiservice,

@manojbadam because metrics is registered in the api server, when performing a namespace deletion, it must query that external api for (namespaced) resources to be deleted (if exist) associated with that namespace. If the extension server isn't available, Kubernetes can't guarantee that all objects have been removed, and it doesn't have a persistent mechanism (in memory or disk) to reconcile later because the root object would have been removed. That happens with any registered api extension service.

@ctron

This comment has been minimized.

Copy link

ctron commented Jan 18, 2019

As I was constantly running into this, I automated this with a small shell script:

https://github.com/ctron/kill-kube-ns/blob/master/kill-kube-ns

It fetches the project, fixes the JSON, starts and properly stops "kubectl proxy", …

Thanks to everyone pointing me into the right direction!

@samvdb

This comment has been minimized.

Copy link

samvdb commented Jan 19, 2019

As I was constantly running into this, I automated this with a small shell script:

https://github.com/ctron/kill-kube-ns/blob/master/kill-kube-ns

It fetches the project, fixes the JSON, starts and properly stops "kubectl proxy", …

Thanks to everyone pointing me into the right direction!

my hero! <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment