Release v1.17.0

[gianlucam76]

🚀 New Features

ClusterHealthCheck: Notification Policy
A ClusterHealthCheck notification can now carry an optional policy that reduces the number of messages sent while a cluster is changing. onlyOnTransition notifies only when a cluster moves between passing and failing, not when the failure message changes while it keeps failing. minInterval sets the minimum time between two notifications for the same cluster, and changes inside the interval are coalesced. failingFor sets how long a cluster must keep failing before the failure is notified, so a failure that ends sooner is never sent, nor is its recovery. What was last delivered is recorded in the NotificationSummary, so the decisions survive a restart. Notifications without a policy behave as before. The failure message now lists resources in a stable order, so the same failures no longer look like a change.
PRs: libsveltos #699, healthcheck-manager #443, sveltos #808

Report Drifted Resources
The ClusterSummary now says which resources drifted. Each feature has a lastDrift with the detection time and the drifted resources (group, kind, namespace, name and, for Helm, the release). It is kept after the drift is corrected and replaced by the next one, and lists at most 20 resources with a truncated flag when more drifted. It is only set for profiles with syncMode ContinuousWithDriftDetection. The dashboard shows the drift history of each profile on a cluster, and sveltosctl has a new command, show drift-history, that lists the drifted resources and when they drifted, filtered by cluster, profile or feature. In pull mode a Helm chart is now re-applied only when it changed or drifted, as in push mode. Before, a change or a drift on one chart re-applied every chart of the profile.
PRs: libsveltos #701, addon-controller #2024, dashboard #199, sveltosctl #454

Kubernetes Events for Failures and Cluster State
Errors that were only in the logs or in a status field are now recorded as events, so they show in kubectl describe and in the events of the namespace. addon-controller records FeatureFailed when a feature fails and is retried. Before, an event was recorded only for failures that are not retried. Classifier records ClassifierFailed and ClassifierLabelConflict, and event-manager records EventTriggerFailed when an EventTrigger fails to deploy or an EventReport cannot be processed. sveltoscluster-manager records an event when a SveltosCluster goes down (AgentUnhealthy in pull mode, ClusterUnreachable in push mode) and ClusterRecovered when it is back. An event is recorded when the failure is new or different, not on every retry, and not for expected states such as a paused cluster. All reasons come from a shared vocabulary in libsveltos. See the upgrade note: the reasons of the events addon-controller already recorded changed.
PRs: libsveltos #703, libsveltos #704, classifier #523, sveltoscluster-manager #390, addon-controller #2032, event-manager #526

Health Monitoring of Deployed Resources
ClusterProfile, Profile and ClusterPromotion have an opt-in healthMonitoring section. When enabled, once every feature of a ClusterSummary is provisioned, addon-controller creates a HealthCheck and a ClusterHealthCheck for it. The default checks cover Deployment, StatefulSet, DaemonSet, Job and PersistentVolumeClaim and report Healthy, Progressing, Degraded or Suspended. Only the resources deployed by the profile are evaluated, including the ones from Helm charts. A ClusterProfile can add its own HealthChecks, or turn the default checks off and use only those. This answers what is unhealthy on a cluster among what Sveltos deployed, separately from validateHealths, which only gates the deployment. The dashboard shows a health badge on each cluster card, a Health column in the Profiles tab and an Unhealthy resources tab with filters. sveltosctl has a new command, show unhealthy-resources, that lists the resources that are not healthy with the profile owning each of them. sveltos-agent caches every object of the monitored kinds in the cluster, about 20 KB of heap per object, whatever the number of profiles that enable monitoring.
PRs: addon-controller #2037, addon-controller #2042, ui-backend #193, dashboard #201, sveltosctl #455

ClusterHealthCheck: ClusterRefs
A ClusterHealthCheck can list clusters by reference in clusterRefs, in addition to selecting them with clusterSelector. The two sets are added together and a cluster that is in both is counted once.
PR: healthcheck-manager #445

Dashboard: Matching Resources per Cluster on an EventTrigger
An EventTrigger can match hundreds of resources in a cluster, and listing all of them made the details page long and slow. Each cluster card now shows the number of matching resources, and clicking it opens a dialog with the list. The list is loaded on demand from a new paginated endpoint, with filters on kind, namespace and name. In ui-backend, /event returns matchingResourcesCount for each cluster instead of the resources array.
PRs: ui-backend #195, dashboard #202

Dashboard: Profile Rollout State
When a profile sets maxUpdate, the clusters not yet reached still run the previous configuration, but they looked provisioned. The profile page now shows a rollout state for each matching cluster: updated, updating, waiting because maxUpdate is reached, paused or not ready. A summary shows the number of clusters in each state and the resolved maxUpdate while a rollout is pending.
PRs: ui-backend #198, dashboard #204

⚙️ Improvements

addon-controller: Stop Polling Profile Deletion While Clusters Are Paused
The ClusterSummary of a paused cluster is not removed until the cluster is unpaused, and a ClusterProfile or Profile being deleted polled for it every 10 seconds, logging an error each time. On a management cluster with a few paused clusters that was about 2,100 errors in 35 minutes. The profile now logs once, waits, and is reconciled again when a paused cluster is unpaused. It keeps polling as before if any remaining ClusterSummary is for a cluster that is not paused.
PR: addon-controller #2023

SveltosCluster with Connection Down Is Not Ready to Be Configured
A SveltosCluster whose connection is down kept Status.Ready true, so it was still processed. It is now skipped and reconciled again when the connection is back. addon-controller no longer tries to collect ResourceSummaries from, or upgrade drift-detection-manager in, a cluster it cannot reach, which removes the repeated compatibility check and i/o timeout errors. Pull mode is unaffected.
PRs: libsveltos #702, addon-controller #2028

ui-backend: Skip the Per-Cluster Access Check When the User Can Get All Clusters
GET /profile checked access with one SubjectAccessReview for each matching cluster. It now first asks once per cluster type whether the user can get all clusters, and if so skips the per-cluster checks. For a profile matching 1,000 clusters that is one call instead of 1,000, and a request went from 238 ms to 1.4 ms on envtest. A user limited to some clusters gets the same result as before, through the per-cluster check.
PR: ui-backend #199

🐞 Bug Fixes

ui-backend: Malformed Bearer Header and OIDC Proxy Diagnostics
An Authorization header with Bearer and no token made ui-backend panic and return a 500. A malformed header or an empty token now gets a 401. When the token of a dashboard user is refused, the log now says what kind of token it was (JWT, Kubernetes ServiceAccount token or opaque), its issuer, audience and expiry, and the host it was sent to, never the token itself. With an OIDC proxy, a ServiceAccount token also gets a hint that the proxy needs token passthrough enabled.
PR: ui-backend #188

ui-backend: ClusterProfile Content Created in the Wrong Namespace
Creating a ClusterProfile from pasted YAML in the dashboard stored its ConfigMap in the hardcoded projectsveltos namespace, so it failed when Sveltos is installed in another namespace. ui-backend now creates it in the namespace it runs in, read from the NAMESPACE environment variable set through the downward API, and falls back to projectsveltos when it is not set.
PR: ui-backend #197

addon-controller: upgradeCRDs Skipped CRDs from Subcharts
With upgradeOptions.upgradeCRDs set, only the crds/ folder of the top-level chart was applied on upgrade, so the CRDs shipped in a subchart kept a stale schema, with nothing logged. This was a regression since the Helm v4 upgrade. CRDs of subcharts are applied again.
PR: addon-controller #2043

addon-controller: Helm Install Overwrote CRDs Already Present in the Cluster
On a fresh install the CRDs of the chart were applied with server-side apply and forced conflicts, silently overwriting, and possibly downgrading, CRDs that already existed. Existing CRDs are now never modified by default. A new installOptions.existingCRDs field selects what to do with an existing CRD that differs from the one in the chart: Skip (default) leaves it and logs it, Fail fails the installation and names the CRDs, and Overwrite keeps the previous behavior. See the upgrade note.
PR: addon-controller #2044

addon-controller: maxUpdate Stalled by Paused Clusters
With maxUpdate set, a paused cluster could keep one of the update slots for as long as it stayed paused, so with maxUpdate 1 no other cluster was updated. This happened in two ways: a cluster paused after it entered the updating list was never removed from it, and a paused cluster that was skipped was still counted as being updated. A paused cluster no longer holds a slot, the rollout moves to the next cluster, and the paused cluster is picked up again when it is unpaused. The same applies to a cluster that is not ready or that no longer exists.
PRs: addon-controller #2051, addon-controller #2052

healthcheck-manager: HealthCheck Left in the Managed Cluster After Its ClusterHealthCheck Is Deleted
When a ClusterHealthCheck is deleted, the HealthCheckReports of a HealthCheck are removed before the HealthCheck itself. If a report was removed by someone else in between, the NotFound error failed the cleanup. Worse, a failure to remove the reports was swallowed, the HealthCheck was left in the managed cluster and the removal was never retried. A report that is already gone is now ignored, and a real failure is returned so the removal is retried.
PR: healthcheck-manager #446

addon-controller: Helm Values Change During an Upgrade Could Be Lost
The hash of the values of a Helm chart was computed again after the upgrade. If a ConfigMap or Secret referenced in valuesFrom changed while the upgrade was running, the stored hash described values the release never received, the following reconciliation found nothing to do, and the release stayed on the old values. The hashes are now taken before the values are read, so a change during an upgrade is applied by the next reconciliation.
PR: addon-controller #2056

addon-controller: Drift Reported While a ResourceSummary Is Processed Was Cleared
After processing a ResourceSummary, addon-controller read it again and cleared its status. A drift reported by drift-detection-manager in between was cleared without being processed, so its feature was not redeployed until a new drift was reported. The status of the processed copy is now reset, and the update fails with a conflict if the ResourceSummary changed, so the next collection processes the drift.
PR: addon-controller #2077

⚠️ Upgrade notes

Event reasons recorded by addon-controller changed. The events it already recorded now use the shared reasons: Sveltos (a feature was deployed) is now FeatureDeployed, FailedNonRetriable is now FeatureFailedNonRetriable and Conflict is now ProfileConflict. The action of these events was ClusterSummary and is now a verb followed by the feature, for example DeployingHelm. The notes are unchanged. Update any alert or filter that matches the old reasons.

Existing CRDs are no longer overwritten on a Helm install. If you relied on an install replacing CRDs that already exist in the cluster, set installOptions.existingCRDs to Overwrite.

[gianlucam76]

Default for existingCrs is Fail

Documentation correctly says Fail is default https://projectsveltos.io/main/addons/helm_charts/#existing-crds-on-install
Code:

addon-controller/api/v1beta1/spec.go

Line 335 in 7c9a164

// +kubebuilder:default:=Fail