ArgoCD doesn't find diff for CRD based resource

[Vladyslav-Miletskyi]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD doesn't find diff, while kubectl diff does.

diff -u -N /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/LIVE-1268004260/actions.summerwind.dev.v1alpha1.RunnerDeployment.runners.ae-nonprod-aws-runner /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/MERGED-4075677893/actions.summerwind.dev.v1alpha1.RunnerDeployment.runners.ae-nonprod-aws-runner
--- /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/LIVE-1268004260/actions.summerwind.dev.v1alpha1.RunnerDeployment.runners.ae-nonprod-aws-runner2022-05-02 23:38:35.000000000 +0200
+++ /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/MERGED-4075677893/actions.summerwind.dev.v1alpha1.RunnerDeployment.runners.ae-nonprod-aws-runner       2022-05-02 23:38:35.000000000 +0200
@@ -5,9 +5,7 @@
   creationTimestamp: "2022-02-03T16:34:01Z"
-  generation: 6637
-  labels:
-    app.kubernetes.io/instance: aws-...st.runners
+  generation: 6638

@@ -46,7 +44,7 @@
       group: ae-kubernetes-runners
-      image: 91.1-ubuntu-20.04-3
+      image:9 1.1-ubuntu-20.04-2
       labels:
diff -u -N /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/LIVE-1268004260/v1.ServiceAccount.runners.ae-runner /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/MERGED-4075677893/v1.ServiceAccount.runners.ae-runner
--- /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/LIVE-1268004260/v1.ServiceAccount.runners.ae-runner        2022-05-02 23:38:35.000000000 +0200
+++ /var/folders/v1/mb2p0hgs27d5cd3jk2d0216w0000gp/T/MERGED-4075677893/v1.ServiceAccount.runners.ae-runner      2022-05-02 23:38:35.000000000 +0200
@@ -5,8 +5,6 @@
   creationTimestamp: "2022-02-03T16:34:01Z"
-  labels:
-    app.kubernetes.io/instance: aws-nonprod-sharedtools-us-west.runners
   namespace: runners
   resourceVersion: "173731962"

To Reproduce

Create a New version of tracked CRD resource. See how it is not applied :/

Expected behavior

When image version changes, it is applied to the repo

Screenshots

Version

argocd version
argocd: v2.3.0+fe42780
  BuildDate: 2022-03-06T06:27:40Z
  GitCommit: fe427802293b090f43f91f5839393174df6c3b3a
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/amd64

Tested on 2.3.0-2.3.3 on 2 separate ArgoCD clusters.

Config

• No additional diff Customizations in Project/App...
• ApplyOutOfSyncOnly is not set

       config:
            users.anonymous.enabled: "false"
            admin.enabled: "false"

            resource.customizations.ignoreDifferences.cert-manager.io_Certificate: |
              jqPathExpressions:
              - '.spec.duration'
              - '.spec.renewBefore'

            resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
              jqPathExpressions:
              - '.webhooks[]?.clientConfig.caBundle'

            resource.customizations.knownTypeFields.actions.summerwind.dev_RunnerDeployment: |
              - field: spec.template.spec
                type: core/v1/PodSpec
              - fields: spec.template.spec
                type: core/v1/EnvVar

            configManagementPlugins: |-
              - name: argocd-vault-plugin
                generate:
                  command: ["argocd-vault-plugin"]
                  args: ["generate", "./"]
              - name: argocd-vault-plugin-helm
                init:
                  command: [sh, -c]
                  args: ["helm dependency build"]
                generate:
                  command: ["sh", "-c"]
                  args: ["helm template . > all.yaml && argocd-vault-plugin generate all.yaml"]
              - name: argocd-vault-plugin-kustomize
                generate:
                  command: ["sh", "-c"]
                  args: ["kustomize build . > all.yaml && argocd-vault-plugin generate all.yaml"]

[Vladyslav-Miletskyi]

Hmmm, isn't it caused it by core/v1/PodSpec?

[Vladyslav-Miletskyi]

When deleted core/v1/PodSpec from resource.customizations.knownTypeFields.actions.summerwind.dev_RunnerDeployment Image change started to cause sync. But also, other diffs started to pop up - resources to Mi to Gi; quotes change in EnvVar, etc. Checking out older versions of argocd...

[Vladyslav-Miletskyi]

switching to

            resource.customizations.knownTypeFields.actions.summerwind.dev_RunnerDeployment: |
              - field: spec.template.spec.resources
                type: core/v1/PodSpec
              - field: spec.template.spec.env
                type: core/v1/PodSpec

helped. But now the question

Should

            resource.customizations.knownTypeFields.actions.summerwind.dev_RunnerDeployment: |
              - field: spec.template.spec
                type: core/v1/PodSpec

ignore a change in image ?

[aabouzaid]

I have the same issue (ArgoCD v2.3.3+0).

[nan-coupa]

I'm seeing a similar issue showing the resource is in sync and not removing a field that should not be present. Desired state for Gloo Upstream CRD:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
...
spec:
...
  healthChecks:
    - eventLogPath: /dev/stdout
      healthyThreshold: 5
      httpHealthCheck:
        path: /
      interval: 30s
      timeout: 5s
      unhealthyThreshold: 2

Current state:

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
...
spec:
...
  healthChecks:
    - eventLogPath: /dev/stdout
      healthyThreshold: 5
      httpHealthCheck:
        path: /
      interval: 30s
      timeout: 5s
      unhealthyThreshold: 2
  ignoreHealthOnHostRemoval: true

It's ignoring the boolean value spec.ignoreHealthOnHostRemoval on the current resource. The documentation only describes how to ignore diffs or how to covert the field to another core/v1/*Spec, I'm didn't find anything in the spec that represents boolean. Any Suggestions?