Support cluster deletion in reconciler

[rJankowski93]

Description

The reconciler has to offer the option to delete a Kyma installation on a cluster. The design approach to use component reconcilers for component-specific operations has to be reused. Each component reconciler has to be able to modify the regular deletion process by custom hooks (pre, post-hooks etc.).

AC

• A cluster can be marked to be deleted (e.g. new cluster state delete_pending)
• Deletion result should be reflected in cluster status (e.g. by adding states like delete_error, delete_ready)
• Deletion of a component has to be be executed by the corresponding component reconciler
• A component reconciler can inject custom logic which can be executed before, instead of the regular deletion logic or after the deletion (pre-delete-, post-delete, and delete hook etc.).
• Component reconcilers are sending heartbeats about their deletion status (e.g. in progress / failures / errors etc.)
• Finally, the deletion of a Kyma created namespace has to happen (but only if no resources created by the customer exist in the namespace).
• A component reconciler can specify selectors / API groups for resources which will be considered by the default-deletion logic and additionally deleted (only if no custom deletion logic is used)
• Quick deletion of resources is supported (deletion by namespace with --force --timeout 0) and finalizers (if some resources are not deletable)

Reasons

Support of a clean deletion process for KYMA.

Notes

[ebensom]

Needed for rma (runtime-monitoring-agent) reconciler, watching

[tobiscr]

Proposal for deletion:

1. Create 3 new states in model.ClusterStatus:

    ClusterStatusDeletePending  Status = "delete_pending"
    ClusterStatusDeleting  Status = "deleting"
    ClusterStatusDeleteError  Status = "delete_error"
    ClusterStatusDeleteReady  Status = "deleted"
    ClusterStatusReconcilePending  Status = "reconcile_pending"
    ClusterStatusReconcileFailed   Status = "reconcile_failed"
    ClusterStatusReconcileDisabled Status = "reconcile_disabled"
    ClusterStatusReconciling    Status = "reconciling"
    ClusterStatusReconcileError Status = "error"
    ClusterStatusReconcileReady Status = "ready"
   

2. Add Delete-Pending state when DELETE-HTTP endpoint is called

     func (i *DefaultInventory) MarkForDeletion(runtimeID string) error {
         clusterState, err := i.GetLatest(runtimeID)
         if err != nil {
                  return err
          }
         _, err = i.UpdateStatus(clusterState, model.ClusterStatusDeletePending)
         return err
     }
   

3. Consider clusters which are marked for deletion when calling cluster.Inventory.ClustersToReconcile() and in cluster.Inventory.ClustersNotReady()

4. Add indicator field to operation to mark it as delete or reconcile operation. This requires changes:

   • define string-type (enum) to indicate the operation
   • add new field to model model.OperationEntity to store the indicator
   • add DB migration file (see ./configs/db/postgres...) + SQLlite DDL (see 00004.....)
   • adjust existing test cases (as a new notNull field was added)
   • update reconciliation.Repository.CreateReconciliation() to consider the new clsuter state delete_pending when creating new operations (set indicator-field properly)

5. Add action information to reconciler.Reconciliation model

6. Extend service.ComponentReconciler struct to allow setting to pre-/post and delete-actions and extend the fluent interface with the addtional With...() functions to allow our uses to set their custom delete-action implementations.

     preDeleteAction  Action
     deleteAction     Action
     postDeleteAction Action
   

7. Inside of the service.runner the defined action has to be considered and the required action (preDelete, delete or post-delete action) has to be called

8. A default delete-action has to be implemented (see email from PB for the expected behaviour)

9. Update service.ReconcilaitionResult.GetResult() to consider whether the cluster should be marked as successfully/non-successfully reconciled or deleted.

[pbochynski]

Additional requirements:

• Generic undeploy reconciler should be used for those components that do not provide own deletion implementation. Consider additional configuration for the component that lists resources that should be deleted (API groups, selector, etc)
• Provide a way to quickly delete all the resources. Command like undeploy --force --timeout 0 should delete all resources from all apigroups with kyma (plus other well-known API groups). Command should remove finalizers if some resources are not deleted.

[rakesh-garimella]

Couple of more requirements:

• if a label is deleted from pod (which would be used as selector by deployment manifest), the the pod should also be deleted in order to prevent orphans (as deletion of label leads to deletion of owner references). Check this issue: [internal Github]
• What should be the behaviour when a label is added to existing pod or service in kyma-system namespace? In current scenario one can append labels to the pods in kyma-system namespace the reconciler (since it does only kubectl apply the label is not removed and would stay as it is)which would mean the deployment manifest is not what we have applied.

[clebs]

Hi @rakesh-garimella,
thanks for the feedback on this issue! As we discussed, to solve this problem we would need a garbage collector feature and I have created an issue for it here: #332.

[clebs]

Fast deletion mentioned in the AC and by @pbochynski will be done in a separate story as agreed in the team.
The new task to track this can be found here: #333

[dariadomagala-sap]

Hello,

I undeployed and then deployed Kyma again, and the installation failed at the cluster essentials step.
The cluster-essentials resources that I have:

NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/cluster-essentials-pod-preset-webhook   ClusterIP   100.69.211.246   <none>        443/TCP   84m

NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cluster-essentials-pod-preset-webhook   0/1     0            0           84m

NAME                                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/cluster-essentials-pod-preset-webhook-568869cdd6   1         0         0       84m