Can't delete namespace; Possible cause is a ThirdPartyResource

[jgensler8]

Issue Report Template

Tectonic Version

1.5.1

Environment

What hardware/cloud provider/hypervisor is being used with Tectonic? AWS

Expected Behavior

When I delete a namespace, namespace will delete after some short period of time (maximum 3 minutes).

Actual Behavior

Resources in namespace are terminated (including an AWS loadbalancer). Namespace never deletes.

Reproduction Steps

1. Create 1.5.1 cluster, no operators
2. Create a sample application in its own namespace.

---
kind: Namespace
apiVersion: v1
metadata:
  name: applications
  labels:
    name: application-namespace
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mydeployment
  namespace: applications
  labels:
    app: mydeployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: mydeployment
    spec:
      containers:
      - name: mydeployment
        image: brndnmtthws/nginx-echo-headers:latest
        ports:
        - name: my-http
          containerPort: 8080
          # hostPort will be a high level port
---
apiVersion: v1
kind: Service
metadata:
  name: mydeployment
  namespace: applications
  labels:
    app: mydeployment
spec:
  type: NodePort
  ports:
  - name: first-thing
    targetPort: my-http
    port: 7777
  selector:
    app: mydeployment

3. Delete the namespace
4. (Optional) Try and create the namespace again

Other Information

I have written a script to find ThirdPartyResources. Although I couldn't find any in the namespace I am trying to delete, I think it ThirdPartyResources may still be at fault.

Logs when service is created after deletion:

$ kubectl apply -f manifests/echoheaders-lb.yaml 
namespace "applications" configured
Error from server (Forbidden): error when creating "manifests/echoheaders-lb.yaml": deployments.extensions "mydeployment" is forbidden: unable to create new content in namespace applications because it is being terminated.
Error from server (Forbidden): error when creating "manifests/echoheaders-lb.yaml": services "mydeployment" is forbidden: unable to create new content in namespace applications because it is being terminated.

Logs from API server:

E0215 19:01:02.356551       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:02.448434       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:03.147809       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:03.949282       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:04.747130       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:05.546798       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:06.357159       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:07.147471       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:07.946490       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:09.311655       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:11.957162       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:17.170309       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:27.492949       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:01:48.055772       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:02:29.105411       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:03:51.109742       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:04:34.096172       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:09:34.103264       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:14:34.093126       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:19:34.098490       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:24:34.101126       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:29:34.100087       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
W0215 19:31:38.269630       6 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
E0215 19:34:34.100316       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:39:34.100515       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"
E0215 19:44:34.102075       6 errors.go:63] apiserver received an error that is not an unversioned.Status: no kind "DeleteOptions" is registered for version "oidc.coreos.com/v1"

I have included most of the logs to the change in timestamp

[mfburnett]

Hey @jgensler8, thanks for filing this issue. I'll let @IvanCherepov take it from here; he should be able to get back to you within a few days and will pull in any other members of our team as needed.

[IvanCherepov]

How did you delete the namespace - from UI or CLI? It's safer option to use kubectl delete.

Can you please provide the outcome of
kubectl get pods,deployments,svc,configmaps,secrets,ing --namespace applications -o wide --show-labels?

From looking at similar k8s issues #19317 and #37278, it looks like this issue has been fixed in 1.5.2.

[jgensler8]

@IvanCherepov looks like the upgrade from 1.5.1 to 1.5.2 has fixed this issue. We've replaced the old cluster because the UI was experiencing issues as well.

Thank you both for the help!