ArgoCD stops syncing/stuck refreshing

[xamroc]

Checklist:

• [-] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [-] I've included steps to reproduce the bug.
• [-] I've pasted the output of argocd version.

Describe the bug

Hi All,

We're also facing the same issue mentioned here: #11458. I opened a new issue as requested by @jsoref

To debug this, we ran an experimental ArgoCD with:

1. a single controller
2. three repo-servers
3. pulling helm from github

From time to time, ArgoCD stops syncing (refreshes looping infinitely) a couple hours at a time. Observations we've made:

1. ArgoCD is restored over time after being stuck a couple hours.
2. The argocd_repo_pending_request_total metric shows a higher value during these times for a single repo-server.
3. The high pending request repo server has stuck processes (refer to Logs).
4. Those processes usually gets delisted from ps but sticks around during high pending request.

To Reproduce

1. Deploy AWS EKS cluster
2. Helm Install https://artifacthub.io/packages/helm/argo/argo-cd/6.7.14
3. Create applications sourcing helm manifests

Expected behavior

ArgoCD should not get stuck refreshing.

Screenshots

Version

  argocd: v2.10.7+b060053
  BuildDate: 2024-04-15T08:45:08Z
  GitCommit: b060053b099b4c81c1e635839a309c9c8c1863e9
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64

Logs

No logs but these processes get stuck during high pending requests.

PID   USER     TIME  COMMAND
170709 999       0:00 /bin/sh -c ssh -i /dev/shm/xxx -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts "$@" ssh -i /dev/shm/xxx -o StrictHostKeyChecking=yes -o UserKnownHostsFile=
170710 999       0:00 ssh -i /dev/shm/22900259 -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts -o SendEnv=GIT_PROTOCOL git@github.com git-upload-pack '<OUR_REPO>.git'

[haooliveira84]

I see the same problem here with v2.11.0!
But, if I set the dynamicClusterDistribution as a false the problem has been resolved.

[jsoref]

tagging @ishitasequeira from #15036