Lazy provisioning of Cluster Addons

[jkroepke]

Hi,

I recently looked into https://projectsveltos.io/main/events/addon_event_deployment/ and had the idea that this concept could maybe be extended.

I like the idea of installing addons based on usage. However, it feels unnatural for end users if they have to create special Kubernetes resources first, e.g. create a magic Kubernetes Secrets to trigger the external-secrets installation.

Instead, I would like to install the CRDs of an addon first. If an end user creates a resource in that CRD API group, the full addon gets installed. If all resources from the addon API group are deleted, the full addon gets deprovisioned.

This concept should already be possible by having a dedicated ClusterAddon for the CRDs and EventSource, and a secondary ClusterAddon for the full controller.

However, it would reduce the complexity here if Sveltos supported lazy deployment as an official mode. This could be part of a ClusterAddon, where the addon controller is able to extract the CRDs from a chart, deploy them, and create an EventTrigger for all discovered CRDs.

The main goal here is to offer a platform that just works. Dev teams should not have to care about which addon they need to order for their cluster. However, installing all addons by default may result in unused compute resources.

[gianlucam76]

Thank you @jkroepke

I like this. A lazy install where only CRDs are installed and as soon as a custom resource is deployed, the full addon gets deployed.

What is your expectation when the last custom resource is deleted? Do we want to remove the full addon and leave only the CRD? In a dynamic environment that might take time.

[jkroepke]

Do we want to remove the full addon and leave only the CRD?

Good question. A Deployment with replicas set to 0 does not hurt. I have no strong opinion about that. What would be the best option.

However, CRDs themselves may also depend on a running controller, especially if Validating or Mutating Webhooks are used, or if CRD Conversion Webhooks are configured. This makes the pattern maybe more complex than it feels at first.

I come more and more to the conclusion that this proposal sounds great, but in reality, the Kubernetes ecosystem can be very complex. There might not be a good golden path to implement or even configure that feature.

[gianlucam76]

Good point on the webhooks. I did not think of that.