Remove finalizers when daemon is stopped on operator exit

[lbach]

When running a daemon; finalizers are added to ensure the kopf daemon is notified if the object is deleted. However, on operator exit finalizers are not removed from the object. Would it make sense to ensure that the call to stop_daemon removes finalizers on some set of DaemonStoppingReason, e.g. on OPERATOR_EXITING?

Currently we have an operator deployed to a namespace and it handles stuff in that namespace. Upon deletion of the namespace we often end up in the eternal Terminating state, because the operator has exited, but hasn't cleared finalizers which results in objects with installed finalizers are stuck as they are waiting for the operator to clear the finalizer.

The issue is somewhat described in https://kopf.readthedocs.io/en/stable/troubleshooting/, however, if the operator upon exiting would remove finalizers then it wouldn't be an issue. And if the operator is started again, daemons will register finalizers once again.

[cpnielsen]

@nolar Is it possible you could give some hint as how to (easily/possibly) manually remove finalizers from objects?

Would we have to iterate on all monitored objects in all monitored namespaces and manually remove the finalizers in a shutdown hook, or is there a better way?

Again, this is not for the general use-case, but for namespace deletion. We happen to use a number of short-lived namespaces for development, and currently there's no good way to shut them down without ending in a stuck state (start namespace deletion -> stop all pods -> try to delete CRDs -> blocked on finalizer -> namespace deletion stalled completely).

[nolar]

Hello. I cannot remember the exact command, but to remove the finalizers manually, you should patch .metadata.finalizers with and empty list or null:

kubectl patch … -p '{"metadata": {"finalizers": []}}'

As for the described scenario, I am not sure what would be a good way. Perhaps, you can exclude your custom resources from the operator's visibility by using when=, labels=, annotations= filters. Once the resource ceases to be visible to the operator (does not match the filters), the finalizer is removed (afair). So, you can do something like this:

def is_not_being_deleted(meta, **_):
    return meta.get('deletionTimestamp') is None

@kopf.daemon(…, when=is_not_being_deleted)
def d(…): …

The operator will stop the daemon and remove the finalizers as soon as that object is marked for deletion.

On the other hand, this should have happened already: the operator already stops daemons for resources being deleted. I am not sure why it did not.

[cpnielsen]

Hi @nolar

So in this case, the problem occurs when the operator and the CRDs it monitors is in the same namespace, and that namespace is deleted (equivalent of kubectl delete namespace xxx). We cannot guarantee the order in which things are deleted, so it often happens that the operator gets deleted first (and hence does not automatically remove finalizers).

So what we want to do is manually just remove all finalizers for everything monitored when the operator is shut down, regardless of whether things are getting deleted or not. We are aware of the caveats (ie. during an operator restart this may cause things to be deleted which should have been handled), but the trade-off for us is worth it.

The example using when= could work for us. We'd want that logic to change using the shutdown handler, so if the "regular" shutdown waits until that handler has finished, that would work for us.

Quick example:

async def is_shutting_down(memo, **_):
    return not memo.shutdown.is_set()  # asyncio.Event()
    
@kopf.daemon(..., when=is_shutting_down):
async def daemon_handler(...):
    ...
   
@kopf.on.cleanup
async def shutdown(memo, **_):
    memo.shutdown.set()
    await asyncio.sleep(10)  # is this necessary to ensure everything is properly shut down?
    # Finalizers are shutdown, continue regular shutdown

[nolar]

Hello.

Would it make sense to ensure that the call to stop_daemon removes finalizers on some set of DaemonStoppingReason, e.g. on OPERATOR_EXITING?

Yes, this totally makes sense. If the only reason the finalizers are added is the daemon (and no other on-change handlers), then the operator should remove them when stopping/exiting.

This logic is currently absent in the code. I am not sure when (and if) I will be able to add it, sorry. But you can contribute the code! ;-)

PS: If there are other reasons (e.g. on-creation/deletion/update handlers), then the finalizers should remain. As a separate change, it might be a good idea to add a CLI option like --ephemeral-finalizers, which will remove all finalizers on exiting regardless of their reasons.

[cpnielsen]

Hello.

Would it make sense to ensure that the call to stop_daemon removes finalizers on some set of DaemonStoppingReason, e.g. on OPERATOR_EXITING?

Yes, this totally makes sense. If the only reason the finalizers are added is the daemon (and no other on-change handlers), then the operator should remove them when stopping/exiting.

This logic is currently absent in the code. I am not sure when (and if) I will be able to add it, sorry. But you can contribute the code! ;-)

PS: If there are other reasons (e.g. on-creation/deletion/update handlers), then the finalizers should remain. As a separate change, it might be a good idea to add a CLI option like --ephemeral-finalizers, which will remove all finalizers on exiting regardless of their reasons.

The --ephemeral-finalizers is pretty much what we're looking for. The "problem" is that even if the operator is long-lived, we want things to be deletable if the operator is stopped (at least in most cases).

I cannot tell exactly if the operator maintains a list of objects with finalizers. If you have a deletion handler on deployments, does it maintain a list of all deployments it has seen and added a finalizer to? Or does it only reactively remove finalizers (when things are being deleted)?

If you can provide some pointers on where and what to look for, we'd be happy to contribute a PR to get this solved 👍

[ArnaudPalgen]

Hello,
I also encountered a problem with persistent finalizers. I had to delete them manually.
Any issue or PR for this feature yet ?