Strategy for removing custom resource of module operator

[pbochynski]

Created on 2023-03-07 by Piotr Bochynski (@pbochynski).

Decision log

Name	Description
Title	Strategy for removing custom resource of module operator
Due date	2023-03-07
Status	Accepted
Decision type	Choice
Affected decisions	-

Context

When Kyma module is disabled or the main custom resource managed by the module operator gets deleted it should wait until all dependent resources are deleted. The module operator should keep the finalizer on the resource and release it only when the last resource was deleted. This is so-called blocking strategy.

In the case of the Kyma cluster termination, the blocking strategy is not optimal. During de-provisioning quick clean-up is much more effective and the operator should drop all the dependencies immediately. This strategy is called cascading delete strategy

Decision

The decision is that blocking strategy is the default for all module operators. Deprovisioning use case should be handled by cascading delete strategy when this feature is implemented.

Consequences

Kyma users are better protected against accidental removal of the module.

[jakobmoellerdev]

Hi there, imo the strategies you mention here are a bit mixed up and do not represent deletion scenarios in Kubernetes.

I quickly want to map your requirements to real Kubernetes scenarios:

The first strategy:

The module operator should keep the finalizer on the resource and release it only when the last resource was deleted

This is the default strategy of any controller and it will act in both foreground cascading and background cascading use cases as mentioned in https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/#set-orphan-deletion-policy

For the second use case:

During de-provisioning quick clean-up is much more effective and the operator should drop all the dependencies immediately. This strategy is called cascading delete strategy

This is actually not a real strategy. Instead this is a combination of "trigger delete but then immediately remove the finaliser". This should not be used in any automation (even cluster deprovisioning). The reason is that one needs to keep parent resources for observability as long as dependencies are still living. "Dropping" can only occur if you accept that the resource is "orphaned". This however is not really faster than a deletion, it just doesn't wait for child resources.

I would really appreciate if you could explain how "dropping" is meant and how it's faster.

[pbochynski]

@jakobmoellerdev Thanks for your comments. I think I used too many shortcuts to keep the decision record concise.

For better understanding let me use the real life example with the Serverless module. It is more complex then the setup with deployments and pods from kubernetes documentation mentioned above.
When you delete a Function CR we have regular cascading delete case. Function owns deployment, deployment owns replicaset which owns pods.
When you delete a module you should remove all APIs that the module introduced. But we do not have owner reference on CRD. Therefore kubernetes will not trigger deletion of functions when you delete serverless module custom resource.

ServerlessCR ---indirectly-creates---> FunctionCRD
user(customer) ---creates(owns)---> FunctionCR
FunctionCR ---owns---> Deployment
Deployment ---owns---> ReplicaSet 
ReplicaSet ---owns---> Pod

The point is that even though the serverlessCR in theory is a root owner of all functions it is not from the Kubernetes perspective. Function CR or FunctionCRD doesn't have owner reference to serverless CR and is not deleted automatically with serverless CR. The decision is to keep it as it is and do not try to trigger deletion of all Function CR when Serverless CR is deleted.
So the cascading deletion will work for function CR, but when Serverless CR will be deleted the finalizer will protect it from deletion until user will not delete the last Function CR (simple way to do it is kubectl delete function --all -A)

That was the blocking strategy. In the cascading strategy we would not wait for user to delete all functions - we could trigger when the ServerlessCR is deleted. The finalizers will work as before - function is deleted only when all deployments are gone, and serverlessCR is deleted when last function was finally removed. The difference is that user has just one command to delete module completely.

Anyway the blocking strategy remains default. For cluster deprovisioning we will implement resource deletion that can deal with the orphan resources (kyma-project/lifecycle-manager#470)

[pbochynski]

The decision is accepted.