Retrieve CRD schemas from the Server so default values can be considered during SSA diff calculation

[leoluz]

Summary

When ServerSideApply is enabled, the new diff logic is currently unable to consider CRD default values. Argo CD should be able to retrieve CRD schemas from Kubernetes server and add to the controller cache so default values can be considered during SSA diff calculation.

Motivation

Currently Argo CD is unable to consider CRD default values while calculating diffs when SSA is enabled. This makes it display the Application as out-of-sync when in fact there is no diff between the desired and live states.

Proposal

We currently cache the GVKParser but this object doesn't have the ability to apply default values. We need to investigate and find a way to apply default values during diff calculation.

[rouke-broersma]

This feature is a must have to consider using server side apply because most CRDs will have some default values and applying ignoreDifferences to all applications for CRD default values is an immense task.

[leoluz]

@rouke-broersma Clarifying things. The current limitation just applies for CRDs that define default values at the schema level.

In openAPIV3Schema the CRD can define a default value for a property like in the example below:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
...
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          someProperty:
            type: string
            default: some-default-value  <-- this

This is the case that will cause problems in the diff logic for SSA.

However, controllers can still define and consider internal default values (only applied in runtime) and in this case there won't be any diff issues in Argo CD. Not every CRD will have default values at the schema level. Argo CD CRDs for example don't define any.

[rouke-broersma]

@leoluz I understand the case where this applies. All our own CRDs define default values at the schema level, and a lot of the applications we deploy do this as well. I still think it's valid to say that most (fine maybe many) CRDs do this.

[asafhm]

This bug pretty much rendered the server-side apply feature useless for me, as I was so hoping it would solve the infamous "CRD too long" issue of kube-prometheus-stack.
Really praying for progress here...

[leoluz]

The new Server-Side Diff feature should be used to cover this use-case