Failing to kill namespaces in Terminating state

[yamaszone]

Thanks for the initiative! I was trying to clean up few zombie namespaces in Terminating state but it doesn't look like doing the clean up correctly. Here's the output I get:

./knsk.sh --delete-all --force

\e[92mKubernetes NameSpace Killer\e[0m

\e[93mChecking if kubectl is configured...\e[0m\e[92m ok\e[0m

\e[93mChecking for unavailable apiservices...\e[0m\e[96m found\e[0m
\e[93m.: Broken -> \e[91mv1beta1.servicecatalog.k8s.io\e[0m...\e[0m\e[91m error\e[0m
.: apiresources deleted, waiting to see if Kubernetes does a clean namespace deletion... ok      

\e[93mChecking for stucked namespaces...\e[0m\e[96m found\e[0m
\e[93m.: Checking resources in namespace \e[91ma\e[0m...\e[0m\e[92m not found\e[0m
\e[93m.: Checking resources in namespace \e[91mb\e[0m...\e[0m\e[92m not found\e[0m
\e[93m.: Checking resources in namespace \e[91mcatalog\e[0m...\e[0m\e[92m not found\e[0m
\e[93m.: Checking resources in namespace \e[91mexp\e[0m...\e[0m\e[92m not found\e[0m

\e[93mChecking for stucked resources in the cluster...\e[0m\e[92m not found\e[0m

\e[93mChecking for orphan resources in the cluster...\e[0m\e[92m not found\e[0m

\e[93mForcing deletion of stucked namespaces\e[0m
\e[93m.: Checking compliance of --force option...\e[0m\e[92m ok\e[0m
\e[93m.: Getting the access token to force deletion...\e[0m\e[92m ok\e[0m
\e[93m.: Starting kubectl proxy...\e[0m\e[92m ok\e[0m
\e[93m.: Checking for resisted stucked namespaces to force deletion...\e[0m\e[96m found\e[0m
\e[93m   > Forcing deletion of a...\e[0m\e[92m ok\e[0m
\e[93m   > Forcing deletion of b...\e[0m\e[92m ok\e[0m
\e[93m   > Forcing deletion of catalog...\e[0m\e[92m ok\e[0m
\e[93m   > Forcing deletion of exp...\e[0m\e[92m ok\e[0m
\e[93m.: Stopping kubectl proxy...\e[0m\e[92m ok\e[0m

\e[93m:: Done in 182 seconds.
\e[0m

Additional info:

$ kubectl version --short
Client Version: v1.14.0
Server Version: v1.14.3

I am using Docker-Desktop v2.1.0.1 on macOS.

Thanks in advance for help!

[thyarles]

Hi @yamaszone,

Thank you for your comment. By the way, sorry for the delay in analyzing your question, I don't know why, but GitHub did not notify me and just now I saw it. Could you please take a look in /tmp folder from a machine that your run the script? There a file named "yournamespacename.json", could you put here the content of it?

Thanks.

[yamaszone]

Hi @thyarles , thanks a lot for following up on this! Here's an example JSON file for you to review:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-12-19T01:57:44Z",
        "deletionTimestamp": "2020-04-09T15:25:26Z",
        "name": "exp",
        "resourceVersion": "7492953",
        "selfLink": "/api/v1/namespaces/exp",
        "uid": "f2e8f518-2202-11ea-a046-025000000001"
    },
    "spec": {
        "finalizers": [

        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

Let me know if you need more information. Thanks in advance for your help!

[thyarles]

Please, could you try to run as

./knsk.sh --delete-all --force --dry-run --no-color

And post the output? You can hide sensitive information, if there any.

[yamaszone]

Hi @thyarles , here's the output:

Kubernetes NameSpace Killer

Checking if kubectl is configured... ok

Checking for unavailable apiservices... found
.: Broken -> v1beta1.servicecatalog.k8s.io... dry-run
   timeout 15 kubectl delete apiservice v1beta1.servicecatalog.k8s.io

Checking for stucked namespaces... found
.: Checking resources in namespace a... not found
.: Checking resources in namespace b... not found
.: Checking resources in namespace catalog... not found
.: Checking resources in namespace exp... not found

Checking for stucked resources in the cluster... not found

Checking for orphan resources in the cluster... not found

Forcing deletion of stucked namespaces
.: Checking compliance of --force option... ok
.: Getting the access token to force deletion... ok
.: Starting kubectl proxy... ok
.: Checking for resisted stucked namespaces to force deletion... found
   > Forcing deletion of a... dry-run
     curl -s -o /tmp/a.json.log -X PUT --data-binary @/tmp/a.json http://localhost:8765/api/v1/namespaces/a/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of b... dry-run
     curl -s -o /tmp/b.json.log -X PUT --data-binary @/tmp/b.json http://localhost:8765/api/v1/namespaces/b/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of catalog... dry-run
     curl -s -o /tmp/catalog.json.log -X PUT --data-binary @/tmp/catalog.json http://localhost:8765/api/v1/namespaces/catalog/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure
   > Forcing deletion of exp... dry-run
     curl -s -o /tmp/exp.json.log -X PUT --data-binary @/tmp/exp.json http://localhost:8765/api/v1/namespaces/exp/finalize                   -H "Content-Type: application/json" --header "Authorization: Bearer ***" --insecure

.: Stopping kubectl proxy... ok

:: Done in 93 seconds.

Let me know if you need any other info.

[thyarles]

Awesome, @yamaszone!

The first thin is figure out why you v1beta1.servicecatalog.k8s.io is broken. You know from it came? Looking in my full working cluster, I don't have it. If you think it is is importante, don't run the command to delete it bellow.

Let's try the commands manually... if all of it works, the error is something about BASH on MacOS.

1. In one shell, let's start the proxy on port 8765 (or other port you want)

KPORT=8765
kubectl proxy -p $KPORT

2. In another terminal, do the commands

kubectl delete apiservice v1beta1.servicecatalog.k8s.io (do this command only if you sure about it)

KPORT=8765 (it must be the same setted before)
BEARER=xxx (paste here your bearer code)

curl -X PUT --data-binary @/tmp/a.json http://localhost:$KPORT/api/v1/namespaces/a/finalize -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/b.json http://localhost:$KPORT/api/v1/namespaces/b/finalize  -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/catalog.json http://localhost:$KPORT/api/v1/namespaces/catalog/finalize -H "Content-Type: application/json" -H  "Authorization: Bearer $BEARER" --insecure

curl -X PUT --data-binary @/tmp/exp.json http://localhost:$KPORT/api/v1/namespaces/exp/finalize -H "Content-Type: application/json" -H "Authorization: Bearer $BEARER" --insecure

Please, let me know if it works. If not, please, take a close look at all output.

[yamaszone]

Hi @thyarles , that worked! Thanks a lot for debugging this and getting rid of annoying namespace termination failures. Cheers!