ArgoCD shows status as synced, but nothing in the kubernetes cluster is updated.

[EStork09]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Application is set to auto-sync. When a git update occurs and the helm chart updates argocd will show out of sync, then will "sync". Once it has "synced" it will show the latest commit is applied, and the events will show that there has been a sync. But in the kubernetes cluster, nothing has changed. The only way to force it is with a hard refresh. Or I believe you wait 4 hours.

To Reproduce

Update Chart to new version, go to argocd, click refresh (not hard refresh).
https://gitlab.com/navenest/kubernetes/apps/go-web-base
https://gitlab.com/navenest/kubernetes/apps/go-web-base/-/commit/e8bf7fdfb36fc4948aa561eb71e1383fcc502e5d

Expected behavior

When argocd says it applied the sync, that it will actually sync.

Screenshots

Applied statement

K8s displaying old version still

Commit that is "synced"

Version

argocd: v2.11.7+e4a0246
  BuildDate: 2024-07-24T10:10:59Z
  GitCommit: e4a0246c4d920bc1e5ee5f9048a99eca7e1d53cb
  GitTreeState: clean
  GoVersion: go1.21.12
  Compiler: gc
  Platform: windows/amd64
argocd-server: v2.11.7+e4a0246
  BuildDate: 2024-07-24T09:33:49Z
  GitCommit: e4a0246c4d920bc1e5ee5f9048a99eca7e1d53cb
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.4+g81c902a
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs

Explore-logs-2024-07-27 14_56_58.txt

[andrii-korotkov-verkada]

Can you share your application's manifest, please?

[EStork09]

project: apps
source:
  repoURL: 'https://gitlab.com/navenest/kubernetes/apps/go-web-base.git'
  path: clusters/rke2
  targetRevision: HEAD
  helm:
    valueFiles:
      - ../../common/values.yaml
      - values.yaml
    version: v3
destination:
  server: 'https://kubernetes.default.svc'
  namespace: go-web-base
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - ServerSideApply=true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m
ignoreDifferences:
  - group: monitoring.coreos.com
    kind: ServiceMonitor
    jqPathExpressions:
      - '.spec.endpoints[]?.relabelings[]?.action'

I have tried serverside apply as I thought maybe changing the sync method would work better, but it didn't seem to make a difference.

[andrii-korotkov-verkada]

It doesn't show things like app name and annotations still. I'm curious whether for example https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#manifest-paths-annotation is used.

[EStork09]

Sorry, that was just a quick grab from the UI.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: clusters/rke2
  creationTimestamp: "2024-07-27T12:29:45Z"
  generation: 3002
  name: go-web-base
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: apps
    uid: 5728fd6a-0227-47d0-a480-2f8748140099
  resourceVersion: "336931335"
  uid: c819f308-30bd-445d-b7a2-0d0a74169ede
spec:
  destination:
    namespace: go-web-base
    server: https://kubernetes.default.svc
  ignoreDifferences:
  - group: monitoring.coreos.com
    jqPathExpressions:
    - .spec.endpoints[]?.relabelings[]?.action
    kind: ServiceMonitor
  project: apps
  source:
    helm:
      valueFiles:
      - ../../common/values.yaml
      - values.yaml
      version: v3
    path: clusters/rke2
    repoURL: https://gitlab.com/navenest/kubernetes/apps/go-web-base.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
      limit: 5
    syncOptions:
    - Validate=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - ServerSideApply=true

[EStork09]

I adjusted argocd.argoproj.io/manifest-generate-paths: clusters/rke2 to argocd.argoproj.io/manifest-generate-paths: .
And that seemed to work? Does the manifest generate paths go before or after the source.path=clusters/rke2

I'll test some more, but I think in my case setting it to . has worked fine, I am not using multiple applications in one repo, so this might have been left over from a previous setup I had.

[andrii-korotkov-verkada]

Since you get values from common/values.yaml, you need generate manifest paths to include both clusters/rke2 and common.

[jblsk]

We experience the same effect after upgrade from v2.10.9 to v2.11.7:
app manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: ./podinfo/overlays/sys-segum
    notifications.argoproj.io/subscribe.on-sync-succeeded.grafana: smurf-team01|podinfo|prod|bcde-aks|sys-segum|pipecleaner
  creationTimestamp: "2023-09-27T12:03:37Z"
  generation: 106251
  labels:
    app.kubernetes.io/name: podinfo-pipecleaner-sys-segum
  name: podinfo-pipecleaner-sys-segum
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: smurf-team01-apps
    uid: 0247a92c-f63d-4a0a-9784-c45d9adf5533
  resourceVersion: "631547887"
  uid: b6e94825-f2ba-4535-8fa7-7cf96f91fa71
spec:
  destination:
    namespace: podinfo
    server: https://my-company.sys.core.intra:6443
  ignoreDifferences:
  - group: argoproj.io
    jsonPointers:
    - /status
    kind: Application
  project: smurf-team01
  source:
    path: ./podinfo/overlays/sys-segum
    repoURL: https://bitbucket.dev.my-company.com/code.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: argocd
  health:
    status: Healthy
  history:
  - deployStartedAt: "2023-09-27T12:03:37Z"
    deployedAt: "2023-09-27T12:03:41Z"
    id: 0
    initiatedBy: {}
    revision: 6f6ef6c9f8064d318eabe4bbe4c00736682fb6d0
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2023-09-27T12:08:48Z"
    deployedAt: "2023-09-27T12:08:49Z"
    id: 1
    initiatedBy: {}
    revision: f34835e4aadad690d73fa657277c988f17e8dae0
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2023-11-06T13:13:31Z"
    deployedAt: "2023-11-06T13:13:34Z"
    id: 2
    initiatedBy: {}
    revision: 72264cdfb14b695d63f67c732eebb209b2f804b7
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-02-12T14:35:05Z"
    deployedAt: "2024-02-12T14:35:07Z"
    id: 3
    initiatedBy: {}
    revision: c54110d1e44afbe3321ee52f72eaba5a2681fdd0
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-03-05T11:41:10Z"
    deployedAt: "2024-03-05T11:41:11Z"
    id: 4
    initiatedBy: {}
    revision: 3eed9f5ef10cd172b797c706ca9dec3f78a22235
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-03-05T12:27:53Z"
    deployedAt: "2024-03-05T12:27:53Z"
    id: 5
    initiatedBy: {}
    revision: 0ad6cc1133eb3dab2cd4e29d0fb04e78e2dcc8f8
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-03-25T13:02:47Z"
    deployedAt: "2024-03-25T13:02:48Z"
    id: 6
    initiatedBy: {}
    revision: c74a95a9f9fc6c3de8a9190d2c8712ccbb3e34a5
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-05-30T11:11:56Z"
    deployedAt: "2024-05-30T11:11:58Z"
    id: 7
    initiatedBy: {}
    revision: 07d576f7c5d9c8aa72b05c49e4f0ae51eedc3949
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  - deployStartedAt: "2024-09-05T11:11:20Z"
    deployedAt: "2024-09-05T11:11:22Z"
    id: 8
    initiatedBy: {}
    revision: de131c1b70ae6bac1fdc6fbda278965408a0b090
    source:
      path: ./podinfo/overlays/sys-segum
      repoURL: https://bitbucket.dev.my-company.com/code.git
  operationState:
    finishedAt: "2024-09-05T11:11:22Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy: {}
      retry: {}
      sync:
        revision: de131c1b70ae6bac1fdc6fbda278965408a0b090
        syncOptions:
        - CreateNamespace=true
        syncStrategy:
          hook: {}
    phase: Succeeded
    startedAt: "2024-09-05T11:11:20Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/podinfo unchanged
        name: podinfo
        namespace: podinfo
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/podinfo unchanged
        name: podinfo
        namespace: podinfo
        status: Synced
        syncPhase: Sync
        version: v1
      revision: de131c1b70ae6bac1fdc6fbda278965408a0b090
      source:
        path: ./podinfo/overlays/sys-segum
        repoURL: https://bitbucket.dev.my-company.com/code.git
  reconciledAt: "2024-09-05T14:33:24Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: podinfo
    namespace: podinfo
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: podinfo
    namespace: podinfo
    status: Synced
    version: v1
  sourceType: Kustomize
  summary: {}
  sync:
    comparedTo:
      destination:
        namespace: podinfo
        server: https://my-company.sys.core.intra:6443
      ignoreDifferences:
      - group: argoproj.io
        jsonPointers:
        - /status
        kind: Application
      source:
        path: ./podinfo/overlays/sys-segum
        repoURL: https://bitbucket.dev.my-company.com/code.git
    revision: de131c1b70ae6bac1fdc6fbda278965408a0b090
    status: Synced

ArgoCD UI screenshots (replicas: 0):

ArgoCD UI sync status (0 replicas):

git commit (replicas: 1):

[jblsk]

metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths:

We discovered the issue in our case was related to our configuration of he above annotation:

1. incorrect use of relative vs absolute path
2. use of the above annotation with unsupported git remote backend (Bitbucket, see related note in docs)

The functionality seems to be restored to normal after the annotation removal from all Application and ApplicatioSet.
If the annotation and related feature is unsupported with Bitbucket as git remote, I'd expect it to have no effect instead of breaking ArgoCD sync functionality completely.