Application Set Controller with ClusterGenerator delete all Application if Cluster Temporally fails to list secret

[murand78]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
If the ApiServer fails to list Custer Secrets All applications are deleted ( and undeployed from target clusters if not set preserveResourcesOnDeletion)

To Reproduce

Create an ApplicationSet with ClusterGenerator, eg:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: rancher-monitoring
  namespace: argocd
spec:
  generators:
      - clusters:
          selector:
            matchLabels:
              environment: production
          values:
            revision: release-1.0
      - clusters:
          selector:
            matchLabels:
              environment: test
          values:
            revision: HEAD
      - clusters:
          selector:
            matchLabels:
              environment: development
          values:
            revision: development
  template:
    metadata:
      name: '{{nameNormalized}}-rancher-monitoring'

Expected behavior
Applications are not deleted if error occurs listing clusters, deletion shall occur only if the cluster list is empty.

Screenshots

Version

$ argocd version --server localhost:8080 --insecure
argocd: v2.11.2+25f7504
  BuildDate: 2024-05-23T13:32:13Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.11.2+25f7504

Logs

Logs of error listing ARGOCD Clusters from argocd-applicationset-controller

	
app_kubernetes_io_name	argocd-applicationset-controller
cluster	loki
container	argocd-applicationset-controller
filename	/var/log/pods/argocd_argocd-applicationset-controller-657d5cc4fd-759g5_6cab55c6-afc4-47eb-8c3c-058b21d99277/argocd-applicationset-controller/0.log
job	argocd/argocd-applicationset-controller-
namespace	argocd
pod	argocd-applicationset-controller-657d5cc4fd-759g5
pod_template_hash	657d5cc4fd
Detected fields
Time	1717335771666
log	"time=\"2024-06-02T13:42:51Z\" level=error msg=\"error generating application from params\" applicationset=argocd/rancher-monitoring error=\"error listing clusters: the server was unable to return a response in the time allotted, but may still be processing the request (get secrets)\" generator=\"{nil &ClusterGenerator{Selector:{map[environment:production] []},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},Values:map[string]string{revision: release-1.0,},} nil nil nil nil nil nil nil nil}\"\n"
stream	"stderr"
time	"2024-06-02T13:42:51.570707233Z"
tsNs	1717335771666269049

Logs of deletion from argocd-notifications-controller ( one per target cluster )

app_kubernetes_io_name	argocd-notifications-controller
cluster	loki
container	argocd-notifications-controller
filename	/var/log/pods/argocd_argocd-notifications-controller-675d57f848-72zxn_57a92c91-ff78-4430-9d99-7089d14b0f07/argocd-notifications-controller/0.log
job	argocd/argocd-notifications-controller-
namespace	argocd
pod	argocd-notifications-controller-675d57f848-72zxn
pod_template_hash	675d57f848
Detected fields
Time	1717335807225
log	"time=\"2024-06-02T13:43:27Z\" level=info msg=\"Trigger on-deleted result: [{[0].BmlU8P6R19BGtNv5gj8udVypmxQ it-office-rancher-monitoring [app-deleted] true}]\" resource=argocd/it-office-rancher-monitoring\n"
stream	"stderr"
time	"2024-06-02T13:43:27.060284451Z"
tsNs	1717335807225691212

Logs of recreation from argocd-notifications-controller ( one per target cluster )

app_kubernetes_io_name	argocd-notifications-controller
cluster	loki
container	argocd-notifications-controller
filename	/var/log/pods/argocd_argocd-notifications-controller-675d57f848-72zxn_57a92c91-ff78-4430-9d99-7089d14b0f07/argocd-notifications-controller/0.log
job	argocd/argocd-notifications-controller-
namespace	argocd
pod	argocd-notifications-controller-675d57f848-72zxn
pod_template_hash	675d57f848
Detected fields
Time	1717335844081
log	"time=\"2024-06-02T13:44:03Z\" level=info msg=\"Trigger on-created result: [{[0].X_5TO4MPCKAyY0ipFgr6_IraRNs it-office-rancher-monitoring [app-created] true}]\" resource=argocd/it-office-rancher-monitoring\n"
stream	"stderr"
time	"2024-06-02T13:44:03.864721666Z"
tsNs	1717335844081874217