Organize Resource Deployment with Tiers

[gianlucam76]

ClusterProfile/Profile instances let deploy add-ons and applications (Helm charts or Kubernetes resources) across a set of managed clusters.

Sometimes there might be a need to tweak deployments for specific clusters (a subset of the original group) within that group.

Previously, creating a new ClusterProfile/Profile targeting a subset of clusters with resources already managed by another profile resulted in conflicts. Sveltos wouldn't allow deployment for those resources.

The concept of tier is introduced to manage deployment priority for resources targeted by multiple configurations.

How it works:

1. Each ClusterProfile/Profile has a new property called tier.
2. This tier value controls the deployment order for resources targeting the same cluster element (e.g., a Kubernetes object or Helm chart).
3. By default, the first configuration to reach the cluster "wins" and deploys the resource.
4. Tiers override this behavior. When conflicts occur, the configuration with the lowest tier value takes precedence and deploys the resource. Higher tier values represent lower priority.
5. The default tier value is 100.

Benefits:

1. Finer control over resource deployment: Tiers allow you to fine-tune deployments within your cluster, especially when multiple configurations manage the same resources.
2. Conflict resolution: Tiers ensure predictable outcomes when multiple configurations target the same resource. The configuration with the most critical deployment (lowest tier) takes priority.

Fixes #305

[gianlucam76]

Here is an example. All managed clusters have labels env:fv
Then a subset has label region:west and another subset region_east.

Initially deploy this ClusterProfile

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: validation-and-monitoring
spec:
  clusterSelector: env=fv
  continueOnConflict: true
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.1.1
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

Prometheus, Grafana and Kyverno are deployed in all clusters.

If now we post this ClusterProfile selecting a subset of managed clusters (the one with region:west):

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
 name: kyverno
spec:
 clusterSelector: region=west
 helmCharts:
 - repositoryURL:    https://kyverno.github.io/kyverno/
   repositoryName:   kyverno
   chartName:        kyverno/kyverno
   chartVersion:     v3.1.4
   releaseName:      kyverno-latest
   releaseNamespace: kyverno
   helmChartAction:  Install

and error will be reported on this

status:
  dependencies: no dependencies
  featureSummaries:
  - failureMessage: cannot manage chart kyverno/kyverno-latest. ClusterSummary validation-and-monitoring-capi-clusterapi-workload
      managing it
    featureID: Helm

With this PR though now we can simply set Tier behaviour. Setting Tier to 50 (so lower than Tier default value of 100) will allow kyverno ClusterProfile to win the conflict and so Kyverno gets upgraded to 3.1.4 only in the region:west clusters

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: kyverno
spec:
  tier: 50
  clusterSelector: region=west
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.1.4
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install

+-----------------------------+---------------+------------+----------------+---------+--------------------------------+------------------------------------------+
|           CLUSTER           | RESOURCE TYPE | NAMESPACE  |      NAME      | VERSION |              TIME              |                 PROFILES                 |
+-----------------------------+---------------+------------+----------------+---------+--------------------------------+------------------------------------------+
| default/clusterapi-workload | helm chart    | prometheus | prometheus     | 23.4.0  | 2024-05-14 12:48:20 +0200 CEST | ClusterProfile/validation-and-monitoring |
| default/clusterapi-workload | helm chart    | grafana    | grafana        | 6.58.9  | 2024-05-14 12:48:31 +0200 CEST | ClusterProfile/validation-and-monitoring |
| default/clusterapi-workload | helm chart    | kyverno    | kyverno-latest | 3.1.4   | 2024-05-14 13:01:55 +0200 CEST | ClusterProfile/kyverno                   |
+-----------------------------+---------------+------------+----------------+---------+--------------------------------+------------------------------------------+

[gianlucam76]

This PR also enhances the message displayed in case of conflict.

For helm chart:

  featureSummaries:
  - failureMessage: |
      cannot manage chart kyverno/kyverno-latest. ClusterSummary validation-and-monitoring-capi-clusterapi-workload managing it.
      cannot manage chart prometheus/prometheus. ClusterSummary validation-and-monitoring-capi-clusterapi-workload managing it.
      cannot manage chart grafana/grafana. ClusterSummary validation-and-monitoring-capi-clusterapi-workload managing it.

For resources

  - failureMessage: |
      Object Deployment:nginx/nginx-deployment currently deployed because of ConfigMap default/nginx.
       Sveltos instance currently deploying this resource: ClusterProfile deploy-resources;
      Object Namespace:/nginx currently deployed because of ConfigMap default/nginx.
       Sveltos instance currently deploying this resource: ClusterProfile deploy-resources;

[egrosdou01]

Very helpful feature for operations. Thanks @gianlucam76!