Applicationset-controller overloads api-server and have memory leak

[shkrid]

We have only 1 ApplicationSet and it kills our k8s control-plane.

ArgoCD v1.8.7+eb3d1fb

applicationset-controller memory usage

api-server go-routines

applicationset-controller logs

time="2021-03-12T01:07:25Z" level=info msg="Starting configmap/secret informers"
time="2021-03-12T01:07:25Z" level=info msg="Configmap/secret informer synced"
time="2021-03-12T01:07:25Z" level=info msg="matched cluster secret" cluster=argocd-eu-west-1
time="2021-03-12T01:07:25Z" level=info msg="matched cluster secret" cluster=argocd-us-east-1
time="2021-03-12T01:07:25Z" level=info msg="generated 2 applications" generator="&{0xc000848f00 0xc00012c000 0xc0007abce0 devops}"
time="2021-03-12T01:07:25Z" level=info msg="unchanged Application" app=eu-west-1-web-autotests appSet=web-autotests
2021-03-12T01:07:25.915Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"eu-west-1-web-autotests\""}
time="2021-03-12T01:07:25Z" level=info msg="unchanged Application" app=us-east-1-web-autotests appSet=web-autotests
time="2021-03-12T01:07:25Z" level=info msg="end reconcile" requeueAfter=0s
time="2021-03-12T01:07:25Z" level=info msg="Starting configmap/secret informers"
2021-03-12T01:07:25.916Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"us-east-1-web-autotests\""}
time="2021-03-12T01:07:26Z" level=info msg="Configmap/secret informer synced"
time="2021-03-12T01:07:26Z" level=info msg="matched cluster secret" cluster=argocd-us-east-1
time="2021-03-12T01:07:26Z" level=info msg="matched cluster secret" cluster=argocd-eu-west-1
time="2021-03-12T01:07:26Z" level=info msg="generated 2 applications" generator="&{0xc000848f00 0xc00012c000 0xc0007abce0 devops}"
time="2021-03-12T01:07:26Z" level=info msg="unchanged Application" app=us-east-1-web-autotests appSet=web-autotests
2021-03-12T01:07:26.124Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"us-east-1-web-autotests\""}
time="2021-03-12T01:07:26Z" level=info msg="unchanged Application" app=eu-west-1-web-autotests appSet=web-autotests
time="2021-03-12T01:07:26Z" level=info msg="end reconcile" requeueAfter=0s
2021-03-12T01:07:26.125Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"eu-west-1-web-autotests\""}


time="2021-03-12T01:10:25Z" level=info msg="Starting configmap/secret informers"
time="2021-03-12T01:10:25Z" level=info msg="Configmap/secret informer synced"
time="2021-03-12T01:10:25Z" level=info msg="matched cluster secret" cluster=argocd-eu-west-1
time="2021-03-12T01:10:25Z" level=info msg="matched cluster secret" cluster=argocd-us-east-1
time="2021-03-12T01:10:25Z" level=info msg="generated 2 applications" generator="&{0xc000848f00 0xc00012c000 0xc0007abce0 devops}"
time="2021-03-12T01:10:25Z" level=info msg="unchanged Application" app=eu-west-1-web-autotests appSet=web-autotests
2021-03-12T01:10:25.907Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"eu-west-1-web-autotests\""}
time="2021-03-12T01:10:25Z" level=info msg="unchanged Application" app=us-east-1-web-autotests appSet=web-autotests
2021-03-12T01:10:25.908Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"us-east-1-web-autotests\""}
time="2021-03-12T01:10:25Z" level=info msg="end reconcile" requeueAfter=0s
time="2021-03-12T01:10:25Z" level=info msg="Starting configmap/secret informers"
time="2021-03-12T01:10:26Z" level=info msg="Configmap/secret informer synced"
time="2021-03-12T01:10:26Z" level=info msg="matched cluster secret" cluster=argocd-eu-west-1
time="2021-03-12T01:10:26Z" level=info msg="matched cluster secret" cluster=argocd-us-east-1
time="2021-03-12T01:10:26Z" level=info msg="generated 2 applications" generator="&{0xc000848f00 0xc00012c000 0xc0007abce0 devops}"
time="2021-03-12T01:10:26Z" level=info msg="unchanged Application" app=eu-west-1-web-autotests appSet=web-autotests
2021-03-12T01:10:26.016Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"eu-west-1-web-autotests\""}
time="2021-03-12T01:10:26Z" level=info msg="unchanged Application" app=us-east-1-web-autotests appSet=web-autotests
2021-03-12T01:10:26.017Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ApplicationSet","namespace":"devops","name":"web-autotests","uid":"a0e6349b-1920-4b8c-8a1d-dc713b04af3f","apiVersion":"argoproj.io/v1alpha1","resourceVersion":"66469235"}, "reason": "unchanged", "message": "unchanged Application \"us-east-1-web-autotests\""}
time="2021-03-12T01:10:26Z" level=info msg="end reconcile" requeueAfter=0s

[jgwest]

I'm able to reproduce and am working on a fix, thanks @shkrid!