Skip to content
This repository has been archived by the owner on Oct 12, 2023. It is now read-only.

Applicationset-controller overloads api-server and have memory leak #155

Merged
merged 1 commit into from
Mar 16, 2021

Conversation

jgwest
Copy link
Member

@jgwest jgwest commented Mar 13, 2021

Fixes #153

This was caused by our creating a SettingsManager (SettingsManager in settings.go in Argo CD) with every call of the ClusterGenerator. Since the settings manager contains a client and 2 listers, each instance was opening sockets and multiple goroutines to handle resource caching with the k8s server. The socket and these connections were what were occupying the leaked memory:

# From SettingsManager
	clientset  kubernetes.Interface
	secrets    v1listers.SecretLister
	configmaps v1listers.ConfigMapLister

The fix is:

  • Create a single instance of SettingsManager (which is what Argo CD does), and query that instance on each cluster generator call.
  • I've also added Argo's stat logger, which outputs the memory usage and # of goroutines every 10 minutes, which should allow us to catch similar issues in the future.

@jgwest jgwest marked this pull request as ready for review March 13, 2021 03:58
@jgwest jgwest requested a review from wtam2018 March 13, 2021 03:58
Copy link
Collaborator

@wtam2018 wtam2018 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
thanks @jgwest !

@jgwest jgwest merged commit 49b3d12 into argoproj:master Mar 16, 2021
@jgwest jgwest deleted the cluster-memory-leak-153 branch June 7, 2021 18:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Applicationset-controller overloads api-server and have memory leak
2 participants