Delete dependency handling PoC using Kyverno

[bobh66]

This is a description of a PoC that uses Kyverno to provide deletion dependency handling. It will block the deletion of a Crossplane Composite or Managed resource that is being used by another resource, as indicated by a label that contains the UID of the depended-on resource.

This information is for reference only - while we did use this PoC for several months in lab testing, the implementation has been changed to Python admission webhooks with a customized kopf framework, which is a lot less resource-intensive solution.

Deletion Dependencies in Crossplane

Background

Crossplane has well-documented issues with leaving orphaned resources in the local cluster and in
the remote cloud provider when a composite resource combines cloud infrastructure and
applications. There are several issues that combine to cause these problems:

• Kubernetes, and Crossplane, default to using Background deletion when resources are deleted, causing all Managed
  and Composite resources to be deleted simultaneously.

• Resources may have implicit dependencies that are not represented in Crossplane but are resolved by eventual
  consistency. For example a Release object may depend on a combination of Cluster and Nodegroup objects, and it will
  eventually be created once the Cluster is available. If those resources are deleted before the Release
  is finished deleting, the Release may be unable to finish the delete process. This causes the ProviderConfig and
  ProviderConfigUsage objects to be orphaned because they are still in use even though the underlying Cluster is gone.

• Cloud provider resources such as Load Balancers, Security Groups, Elastic Network Interfaces and DNS records
  may be orphaned because they were allocated by a Helm chart or other resource that was not properly cleaned up,
  and the cloud resources are not directly visible to Crossplane.

The first issue was resolved in the 1.10 release of Crossplane with the addition of the compositeDeletePolicy
attribute on the Claim. When this attribute is set to Foreground it causes Crossplane to delete the Composite
resource using the Foreground propagation policy, which causes a "bottom up" deletion path. That is
a requirement for the next step in providing support for deletion dependencies, which is "usage tracking" or manual
specification of delete dependencies between resources.

NOTE: the Claim MUST have the compositeDeletePolicy set to Foreground for the deletion to work
properly using the methods described below. If there is no Claim and the Composite is being deleted manually the
kubectl delete command must include --cascade=foreground to trigger Foreground Deletion.

If the default Background policy is used none of these changes will have any effect.

Disclaimer

Everything described here should be considered as Proof of Concept only. This is NOT production grade and it
WILL cause problems in varied and unusual ways. There are better ways to do this but they require time
and development effort. Kyverno enables this functionality through policies but it can shut down your cluster when not
configured properly.

Overview

It is possible to control the deletion of Crossplane composite/managed resources
using a combination of a Custom Resource, Foreground Cascading Deletion, Kyverno policies,
and labels on resources in a Composition. The labels
specify dependencies between resources that Crossplane cannot identify, and a Kyverno policy uses these labels
to create instances of a Custom Resource to keep track of the dependencies. A second set of Kyverno policies is used to
monitor all DELETE API requests, look for existing dependencies on the object being deleted, and reject the API
call if there is a dependency. Kubernetes will continue to attempt to delete the resource until all
dependencies have been deleted and the policy allows the resource to be deleted.

Details

Custom Resource

A Custom Resource is used to track the dependencies between Crossplane resources. It can be as simple as:

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: dependencies.myapis.com
spec:
  scope: Cluster
  group: myapis.com
  names:
    kind: Dependency
    plural: dependencies
    singular: dependency
  versions:
    - name: v1
      served: true
      storage: true
      subresources: { status: { } } 

Note that the resource has no spec attributes, as metadata annotations and labels are used to store the necessary
information.

Kyverno Setup

Kyverno is being used to implement policies that control the creation of Dependencies and the deletion of
resources that have dependencies. In this scenario Kyverno is only handling the policies that are defined here, so the
deployment and it's webhook configuration have been customized to be specific to this task.

NOTE that Kyverno 1.8.1 (helm chart 2.6.1) is the latest version of Kyverno that supports the policies defined below.
Changes were made in Kyverno 1.8.2 that prevent the policies from executing as desired.

Helm Chart Deployment

Kyverno should be deployed with the following settings:

args:
- --autoUpdateWebhooks=false
- --admissionReports=false
replicaCount: 3

This will prevent Kyverno from maintaining (but not creating) the webhooks that send admission requests to the Kyverno
services and deploy 3 Kyverno pods. The default Kyverno webhook configurations are modified below.

ClusterRole

Define a Cluster Role that allows Kyverno to create Dependency objects:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: kyverno
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/name: kyverno
  name: kyverno:dependencies
rules:
- apiGroups:
  - myapis.com
  resources:
  - dependencies
  verbs:
  - '*'

Webhook Configuration

Kyverno will deploy a number of webhook configurations by default and two of them need to be updated to restrict
the number of admission requests that are sent to Kyverno.

Kyverno creates a MutatingWebhookConfiguration named kyverno-resource-mutating-webhook-cfg that is modified
to look like this:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  labels:
    webhook.kyverno.io/managed-by: kyverno
  name: kyverno-resource-mutating-webhook-cfg
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: <leave this field alone>
    service:
      name: kyverno-svc
      namespace: kyverno
      path: /mutate/ignore
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: mutate.kyverno.svc-ignore
  namespaceSelector: {}
  objectSelector:
    matchExpressions:
    - key: crossplane.io/composite
      operator: Exists
    - key: dependsOnUid
      operator: Exists
  reinvocationPolicy: IfNeeded
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    resources:
    - '*/*'
    scope: '*'
  sideEffects: NoneOnDryRun
  timeoutSeconds: 30

The major changes are the addition of the objectSelector.matchExpressions and the removal of all operations except
CREATE. The timeoutSeconds has also been changed to 30 but that is not required. This configuration sends all API CREATE (POST) requests to Kyverno IF they have a crossplane.io/composite label
AND a dependsOnUid label. That will significantly reduce the number of requests sent to Kyverno.

Kyverno also creates a ValidatingWebhookConfiguration object named kyverno-resource-validating-webhook-cfg which
are modified to look like this:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    webhook.kyverno.io/managed-by: kyverno
  name: kyverno-resource-validating-webhook-cfg
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    caBundle: <don't modify this field>
    service:
      name: kyverno-svc
      namespace: kyverno
      path: /validate/ignore
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: validate.kyverno.svc-ignore
  namespaceSelector: {}
  objectSelector:
    matchExpressions:
    - key: crossplane.io/composite
      operator: Exists
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - DELETE
    resources:
    - '*/*'
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30

The major changes are the addition of the objectSelector.matchExpressions and the removal of all operations except
DELETE. The timeoutSeconds has also been changed to 30 but that is not required. This configuration will only send DELETE requests to Kyverno IF the resource has a crossplane.io/composite label.
This prevents the configuration from affecting any Kubernetes resource that is NOT a Composite or Managed resource
derived from a Composite/Composition.

Policies

Now that Kyverno is running and specific API requests are being routed to it, it needs policies to execute.

The first policy executes on resource creation and will create a Dependency object when the new resource
has the dependsOnUid label.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-dependency
spec:
  background: false
  failurePolicy: Ignore
  rules:
  - generate:
      apiVersion: myapis.com/v1
      data:
        metadata:
          labels:
            dependsOn: '{{ request.object.metadata.labels.dependsOnUid }}'
            crossplane.io/composite: '{{ request.object.metadata.labels."crossplane.io/composite" }}'
          annotations:
            kyverno.io/generated-by-resource: '{{ request.resource.resource }}'
            kyverno.io/generated-by-apiversion: '{{ request.object.apiVersion }}'
          ownerReferences:
          - apiVersion: '{{ request.object.apiVersion }}'
            blockOwnerDeletion: false
            kind: '{{request.kind.kind}}'
            name: '{{request.name}}'
            uid: '{{request.object.metadata.uid}}'
      kind: Dependency
      name: '{{ request.object.metadata.uid }}'
    match:
      any:
      - resources:
          kinds:
          - '*'
          selector:
            matchLabels:
              crossplane.io/composite: "?*"
              dependsOnUid: "?*"
    name: check-for-dependency-annotation
    preconditions:
      all:
      - key: '{{ request.operation }}'
        operator: Equals
        value: CREATE
      - key: "{{ request.object.metadata.labels.dependsOnUid || '' }}"
        operator: Equals
        value: "?*"
  validationFailureAction: enforce

This policy looks for incoming CREATE requests for resources that have the crossplane.io/composite and dependsOnUid
labels, and if those match/preconditions succeed, it creates (generates) a Dependency object that keeps track of
the dependency between the resource that is being created and the resource identified by the UID in the label.
The Dependency object is configured to have the creating resource as it's ownerReference so that it will be
deleted automatically when the parent resource is deleted.

The second policy executes on resource deletion and has two components.

The first component will look for any existing Dependency objects that are referring to the resource that is
being deleted. If any Dependency objects are found for that resource, the DELETE request
is rejected (409 Conflict).

The second component looks for DELETE requests on Dependency objects and determines whether the owner of the
Dependency still exists. If the owner still exists the DELETE request is rejected. This prevents the Dependency
object from being deleted by Foreground Cascading Deletion before the depending object has been deleted.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-delete-on-dependencies
spec:
  background: false
  failurePolicy: Ignore
  rules:
  - context:
    - apiCall:
        jmesPath: "items[?metadata.labels.dependsOn == '{{ request.oldObject.metadata.uid }}'] | length(@)"
        urlPath: /apis/myapis.com/v1/dependencies
      name: dependencyCount
    match:
      any:
      - resources:
          kinds:
          - '*'
          selector:
            matchLabels:
              crossplane.io/composite: '?*'
    name: check-for-dependency
    preconditions:
      all:
      - key: '{{ request.operation }}'
        operator: Equals
        value: DELETE
    validate:
      deny:
        conditions:
          any:
          - key: '{{ dependencyCount }}'
            operator: GreaterThan
            value: 0
      message: Block deletion when there is a dependency
  - context:
    - apiCall:
        jmesPath: "items[?metadata.labels.dependsOnUid == '{{ request.oldObject.metadata.labels.dependsOn }}'] | length(@)"
        urlPath: /apis/{{ request.oldObject.metadata.annotations."kyverno.io/generated-by-apiversion" }}/{{ request.oldObject.metadata.annotations."kyverno.io/generated-by-resource" }}
      name: dependencyCount
    match:
      any:
      - resources:
          kinds:
          - 'Dependency'
    name: Check for depending resource
    preconditions:
      any:
      - key: '{{ request.operation }}'
        operator: Equals
        value: DELETE
    validate:
      deny:
        conditions:
          any:
          - key: '{{ dependencyCount }}'
            operator: GreaterThan
            value: 0
      message: Block deletion of the dependency until the dependant resource is deleted
  validationFailureAction: enforce

Putting It All Together

The last step is to add the dependsOnUid label to the resources in the Composition that need to be deleted
in a specific order.

For example in a Composition that deploys a Composite resource to create an EKS cluster, and then deploys a
provider-helm Release managed resource into that cluster it may be necessary to prevent the Cluster
composite object from being deleted until the Release is deleted. That dependency can be specified in the
Composition.

First save the UID of the resource that is going to be depended-on:

  - name: clustercomposite
    base:
      apiVersion: myapis.com/v1alpha1
      kind: XCluster
      spec: {}
    patches:
    - type: ToCompositeFieldPath
      fromFieldPath: metadata.uid
      toFieldPath: metadata.annotations.clusterUid
    - other patches as needed

Now patch that UID value into the dependsOnUid label for the Release object:

  - name: helmchart
    base:
      apiVersion: helm.crossplane.io/v1beta1
      kind: Release
      spec:
        parameters: {}
    patches:
    - type: FromCompositeFieldPath
      policy:
        fromFieldPath: Required
      fromFieldPath: metadata.annotations.clusterUid
      toFieldPath: metadata.labels.dependsOnUid
    - other patches as needed

When the Composition is executed by Crossplane, the Release object will wait until the UID of the XCluster
resource is available, and then will create the Release. Kyverno will see the CREATE request and the dependsOnUid
label and will create a Dependency object with the associated information. kubectl lineage will show the
relationship between the Release and the Dependency objects.

NOTE that the Claim MUST have the compositeDeletePolicy set to Foreground for the deletion to work
properly. If the default Background policy is used the Dependency resources will have no effect.

When the Claim is deleted, Crossplane will see the compositeDeletePolicy is set to Foreground and will request
deletion of the associated Composite resource using Foreground propagation. Deletion of the Composite will trigger
deletion of all of the resources owner by the Composite, in this case the XCluster composite and the Release
resource.

Deletion of the XCluster will be rejected by Kyverno because the Dependency object exists. Kubernetes will retry
the deletion until it is allowed to succeed by the policy. At the same time the deletion of the Release object
proceeds normally as it has no Dependencies. Kubernetes will request deletion of the Dependency object that is
owned by the Release but Kyverno will block it until the Release itself has been deleted.

Once the Release and the Dependency objects have been deleted, the next attempt by Kubernetes to delete the
XCluster object will succeed, and the rest of the deletion process will continue.

Note that the deletion retries by Kubernetes use an exponential backoff with a maximum of 1000 seconds, so there
may be (long) pauses observed between the deletion of the Dependency object and the deletion of the resource
that was depended-on.

kubectl lineage is an excellent way to track the deletion process and to observe the impact of the Dependency
resources on the process.

[nstogner]

When applied to the "Crossplane managing EKS" use-case this would tackle the deletion ordering of child-cluster resources that were created by Crossplane, but not child-cluster resources that might have been created outside of Crossplane, correct? (for example: admin stands up EKS cluster claim, then app devs apply some Service type: LoadBalancer's into child cluster using a pipeline - this is not the target use-case correct?).

[bobh66]

Correct - this is only targeted at resources that Crossplane knows about/manages, and the logical deletion dependencies that the composition author knows about (Release depends on Cluster, etc) but that Crossplane itself cannot derive. Reconciliation resolves the logical creation dependencies, because the Release will keep trying to install until the Cluster becomes ready, but there is no method to automatically derive the deletion dependencies - there is no record of the creation process that would result in a dependency graph usable for deletion.

External resources that Crossplane doesn't know about are not in scope for this effort.