Bug: ArgoCD apps get stuck refreshing and not picking up new Git commits when switching from destination server url to destination cluster name and project destinations to match at the same time

[HariSekhon]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD apps gets stuck refreshing and don't pick up new Git commits when changing their destination cluster from server url to name and changing the project destination server at the same time.

Expected behavior

Expected the apps to sync as both the destination in the app and the destination in the project were changes at the same time

To Reproduce

Change an app argoproj.io/v1alpha1 / Application from this:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  destination:
    server: https://x.x.x.x

to this

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  destination:
    name: some-cluster

while also changing the project argoproj.io/v1alpha1 / AppProject from this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
spec:
  destinations:
   - namespace: whatever
     server: https://x.x.x.x

to this:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
spec:
  destinations:
   - namespace: whatever
     server: some-cluster  # must match the NAME field in the /settings/clusters page

I restarted every single componet of ArgoCD but it didn't clear it.

Hard refreshes didn't work.

New Git commits were no longer being picked up by the apps.

Recovery Workaround

The only thing I could find to resolve it was this series of steps:

1. add the old destinations back into the project
2. hit Sync in ArgoCD to force the apps to sync with their old destination refs on their old stuck git commit in argocd
3. wait for the the apps to sync
4. Refresh + Sync to the latest commit
   5. remove the old destination server urls from the project

Looks like removing the old destination IP URLs from the Argo project.yaml causes this issue in all apps even after they are all force sync'd from the latest git commit. I can't remove the old destination URLs without breaking the Argo apps.

Non-cascading delete + recreation of the app doesn't seem to work either, it continues retry attempt #4 of the stuck failing sync...

Version

$ argocd version
argocd: v2.7.1+5e54351
  BuildDate: 2023-05-02T16:54:25Z
  GitCommit: 5e543518dbdb5384fa61c938ce3e045b4c5be325
  GitTreeState: clean
  GoVersion: go1.19.8
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.6.7+5bcd846
  BuildDate: 2023-03-23T14:57:27Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

[nakamorichi]

Likely related: after upgrade to 2.7.1 server name seems broken; getting Unable to load data: error getting destination cluster at the UI with in-cluster name but switching to URL fixes the error.

[nakamorichi]

Is this fixed by #13564? Haven't had time to check yet.

Maybe also related to #13715 #13770 #13857 ?

[dstevensson]

Just like to add that I ran into this bug but that it is no longer present for me after moving from 2.7.4 to 2.7.14