Implement Server-Side Diff in Argo CD controller

[leoluz]

Summary

Introduce a new diff option to enable calculating diffs using server-side apply dryrun.

Motivation

Argo CD controller implements a diff logic comparing desired and live state to define if the resource is out of sync. With the introduction of Server-Side Apply as a new sync option in 2.5, a new diff logic was implemented trying to reproduce what Kubernetes does while calculating patches during SSA syncs. The new diff logic introduces new challenges to Argo CD controller as it has to keep a local cache of all CRD schemas from all clusters it syncs to. There are different issues with this approach but the main problems are:

• Schemas returned from k8s API don't have default values:
  • Retrieve CRD schemas from the Server so default values can be considered during SSA diff calculation #11139
• API version convertors aren't all available in the cached GVK Parser:
  • ServerSideApply fails with "conversion failed" #11136
• CRD schema updates might be out of sync for a moment in the cluster cache

Addressing the root of those issues will require a deeper dive in Kubernetes Server-Side Apply logic to try to bring that same functionality to Argo CD controller. This may lead to inconsistencies and will likely require a relatively big engineering effort to stabilize the feature.

Proposal

1. Introduce a new diff option to enable Server-Side Diff
2. Server-Side Diff will execute a server-side apply in dryrun mode
3. Argo CD controller will cache the SSA response to avoid hitting kube-api too often
4. SSA response state will be used to compare with live state

This will also address the current limitation with admission controllers as mutating webhooks are only executed in the cluster.

[leoluz]

Will be available as part of Argo CD 2.10-rc1