2.6 multi-source apps: Changes to K8s resources are not picked up automatically

[zenitraM]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When creating an application that depends on the new multiple source support coming on 2.6, changes to cluster resources are not detected properly.

To Reproduce

Create two applications that are the same, except for one being single-sourced and the second being multi-sourced (injecting a no-op Helm values file):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-cluster-test-single-source
  namespace: argocd
spec:
  destination:
    namespace: test-single-source
    server: https://kubernetes.default.svc
  project: default
  source:
    path: chart/
    repoURL: https://github.com/zenitram/argocd-test-helm-chart
    targetRevision: 7d0257a
  syncPolicy:
    automated:
      prune: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m0s
      limit: 5
    syncOptions:
    - CreateNamespace=true

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-cluster-test-multiple-source
  namespace: argocd
spec:
  destination:
    namespace: test-multiple-source
    server: https://kubernetes.default.svc
  project: default
  sources:
  - repoURL: https://github.com/zenitram/argocd-test-values-file
    targetRevision: main
    ref: values
  - path: chart/
    repoURL: https://github.com/zenitram/argocd-test-helm-chart
    targetRevision: 7d0257a
    helm:
      ignoreMissingValueFiles: true
      valueFiles:
      - $values/values.yaml
  syncPolicy:
    automated:
      prune: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m0s
      limit: 5
    syncOptions:
    - CreateNamespace=true

• Wait for both apps to become synced
• Watch for ArgoCD events (I used unbuffer kubectl get event -n argocd --watch-only=true |ts)
• Once the sync is done, delete the test-configmap CM from both namespaces:

kubectl delete -n test-single-source cm test-configmap; kubectl delete -n test-multiple-source cm test-configmap

• The deletion is only picked up by the test-single-source app:

Feb 06 19:09:01 0s          Normal   ResourceUpdated      application/test-cluster-test-single-source       Updated sync status: Synced -> OutOfSync

On the test-multiple-source app, on the UI you need to click Refresh to actually see the removal happen.
Even after that, clicking Sync on the UI recreates the configmap but keeps the state to Unsynced until I hit Refresh again.

Expected behavior

The removal of a resource should cause ArgoCD to pick up automatically the same way it does with a single-sourced one.

Version

I've tested with both 2.6rc6 and a custom build from commit 5b3152c (#12217) to check if that PR is related -- I suspect it is not, since this one sounds more about changes in the managed resources rather than on the Git repo 😅 .

[noahburrell0]

I'm seeing the same problem with multi-source apps on 2.6.0. It looks like reconciliation is occurring on webhook events, but that is about it. Reconciliation does not occur on the default 3 minute interval (or any other interval if I set timeout.reconciliation), it does not occur when a resource is manually deleted or modified through the web interface, and as it was pointed out there is no reconciliation following a sync.

It seems like reconciliation is only ever occurring on webhook events or by manually refreshing the application.

[sfynx]

Experiencing the same issue. Syncing or manipulating already refreshed resources of a multi-source application works just fine, and its related progress is shown at the top (waiting for resources to become healthy, hooks, etc.). However, no resource status is being updated in the tree view until a manual refresh is done.

When creating new resources, everything stays missing and is reported as not being part of the app when trying to inspect things, and any manipulation of k8s resources (including from the UI itself) is not being reflected in the UI until the next manual refresh.

[cjc7373]

I think this was already fixed in #12576 and released in 2.6.3

[zenitraM]

Yes, this was solved already on 2.6.3.