Reconcillation loop when using orphanedResources monitorng + ApplicationSet on ~160 namespaces

[keskad]

Hi! :)

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
Enabling orphanedResources monitoring for larger number of namespaces makes ArgoCD application-controller to use 1.5-2.0 cores instead of ~0.25.

It looks like the application-controller is stuck in a reconcillation loop. I tried bumping the processors values, without luck.

Use case

I have ApplicationSet that for every namespace (160 namespaces) is applying Network Policies, Resource Quotas and Limit Rages. The resources for every namespace are plain YAML files. 160 directories = 160 Applications = resources for 160 namespaces.

Every directory has about 3-8 simple resources of those 3 types.

I would like to know if somebody made a Network Policy, Resource Quota or Limit Range by hand when viewing the Application for given namespace.

ArgoCD configuration using operator:

appSync: 5m
    processors:
      operation: 30
      status: 60

To Reproduce

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
    name: limits
    namespace: cluster-config
spec:
    orphanedResources:
        warn: false
    namespaceResourceWhitelist:
        - group: 'networking.k8s.io'
          kind: 'NetworkPolicy'
        - group: ''
          kind: 'ResourceQuota'
        - group: ''
          kind: 'LimitRange'
        - group: 'network.openshift.io'
          kind: 'EgressNetworkPolicy'
    clusterResourceWhitelist: []
    destinations:
        - namespace: '*'
          server: 'https://kubernetes.default.svc'
    sourceRepos:
        - '*'

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  labels:
    argocd: cluster-config-importer
  name: limits
  namespace: cluster-config
spec:
  generators:
    - git:
        directories:
          - path: limits/by-ns/name/*
          - exclude: true
            path: limits/by-ns/name/*-prod
        repoURL: 'xxxxxxxxxxxxxxxx'
        revision: main
  template:
    metadata:
      name: 'limits-{{path.basenameNormalized}}'
      namespace: cluster-config
    spec:
      destination:
        namespace: '{{path.basenameNormalized}}'
        server: 'https://kubernetes.default.svc'
      project: limits
      source:
        path: 'limits/by-ns/name/{{path.basenameNormalized}}'
        repoURL: 'xxxxxxxxxxxxxxxxxxxxxxx'
        targetRevision: main
      syncPolicy:
        automated:
          prune: true
          selfHeal: false
        syncOptions:
          - ApplyOutOfSyncOnly=true
          - CreateNamespace=false

To fix (workaround)

Disable orphanedResources section (should be not present at all) in kind: AppProject, then delete pod for application-controller.

Expected behavior

I expect to be able to use orphanedResources monitoring together with namespaceResourceWhitelist for discovering if somebody not created Network Policies etc. by hand.

Screenshots

Load diagram:
At the start of diagram (about 15:30) the "orphanedResources" is enabled. It is fine till 09:00 where probably first synchronization happens.

Version

v2.3.3+07ac038

Logs

Nothing. Just info messages looping in application-controller, looking normally, no warnings, no errors. Currently does not have those logs.