[Ansible-Operator] - Unable to deleted namespace when CR of an Ansible operator has a Finalizer

[camilamacedo86]

Bug Report

• Unable to deleted namespace when CR of an Ansible operator has a Finalizer
• It doesn't occur with Golang operators
• It occurs always when has a finalizer and we try to delete the namespace.
• The Ansible operator is able to remove the metadata of the finalizer when deleting is directly made for the CR and not the namespace. (oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml)
• The issue occurs because when we run oc delete project <namespace> the operator has been deleted before the CR. (the issue is with the cascade/link of the resources generated )

What did you do?

• Checked it with the Memcached example: https://github.com/operator-framework/operator-sdk-samples/tree/master/ansible/memcached-operator
• Added just the finalizer:

---
- version: v1alpha1
  group: cache.example.com
  kind: Memcached
  role: /opt/ansible/roles/memcached
  finalizer:
    name: finalizer.cache.example.com

• Create an namespace and apply the example.

$ oc new-project memcached

$ kubectl create -f deploy/crds/cache_v1alpha1_memcached_crd.yaml
$ kubectl create -f deploy/service_account.yaml
$ kubectl create -f deploy/role.yaml
$ kubectl create -f deploy/role_binding.yaml
$ kubectl create -f deploy/operator.yaml
$ kubectl create -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

• Try to delete the namespace

$ oc project delete memcached

What did you expect to see?
The CR + Operator + Namespace be deleted with success.

What did you see instead? Under which circumstances?
The namespace is marked to be deleted, the operator is deleted, but the CR is not which not allows the namespace to be deleted as well and is hanging it.

• Reason: The operator has been deleted before the CR then it cannot remove the finalizer metadata from it which causes the bug.

• Workarround: manual deletion of the finalizer metadata from the CR which would be made by the operator if it was not deleted first. E.g oc patch memcached example-memcached -p '{"metadata":{"finalizers": []}}' --type=merge

OR

• Delete the CR before deleting the namespace for the operator be able to remove the finalizer metadata.oc delete -f deploy/crds/cache_v1alpha1_memcached_cr.yaml

Environment

• operator-sdk version: 0.8.1
• go version: go version go1.12.5 darwin/amd64
• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:34:27Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-05-02T11:52:09Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: Minishift
• Are you writing your operator in ansible, helm, or go? Ansible

Additional context
Following the images to illustrate the bug.

• Namespaced marked to be deleted and hanged by the CR with finalizer.

• See that the operator was deleted before the CR.

• See that the CR still there with the finalizer metadata which should be removed by the operator.

NOTE: Issue opened in order to make clear the scenario/bug raised in #1493

[camilamacedo86]

Hi @estroz,

I closed #1503 because I think that it had too many comments that will be not helpful for we are able to address this scenario.

Unfortunately, shows that my comments/descriptions there were not clear enough for @rcernich and @jmazzitelli and/or they are facing another issue which should be reported separately.

Please, could you add here in this on the respective label? Really thank you for your time and support.

[rcernich]

@camilamacedo86, sorry for the confusion. This issue is not the problem I was describing, but sounds like it is the issue that @jmazzitelli was having. If you want me to create an issue for the problem I was describing, I can do that. As I mentioned in the other issue, the problem I was describing was very difficult to track down the root cause. Once again, sorry for the confusion.

[camilamacedo86]

Hi, @rcernich no problem at all

IMHO: It is better we have one issue for each issue :-) How much more clear we be able to provide the info better, easier and faster can be the fix 👍 So, please feel free to raise your scenario in a new issue.

Thank you in advance for your understanding and collaboration.

[robbie-demuth]

I believe this is also a problem for Helm-based operators. Custom resources created from Helm CRDs have a uninstall-helm-release finalizer. If the custom resource is namespace-scoped, deleting the namespace deletes the controller at which point the finalizer on the custom resources in the namespace cannot be handled - causing the namespace to get stuck in a terminating state

[shawn-hurley]

I believe it is a problem with all operators. Once the delete on the namespace is called, there is nothing that the operator could do.

[camilamacedo86]

HI @shawn-hurley it worked 100% in the Go operator ones. Note that the diff between both is that when it works in Go the operator pod is not removed before the CR/CRD be deleted. I believe that it is the root cause, the operator has been deleted before the CR then it cannot remove the finalizer metadata from it which causes the bug in the ansible ones.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[camilamacedo86]

/remove-lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.