fix: Fixes sharding placement algorithm and allows development of alternative algorithms #13018

akram · 2023-03-27T07:45:19Z

Defines a new API based on distributionFunction that allows developer to introduce other distribution and placement algorithms
Defines a new algorithm named hash that uses db.clusterList index as the sharding placement parameter
Maintain the previous algorithm as legacy and keep it as default as it works with integers as strings of clusters uid. In pratice, uid are uuid which makes the default algorihm not work well.
Fixes the argocd admin shard status to use the distributionFunction and hence produce the same results as the actual placement.
Fixes CodeQL issue related to insecure string to int conversion
Adds log messages

Fixes #9633 #11537 #13136 #11537 #13226

Checklist:

Please see Contribution FAQs if you have questions about your pull-request.

codecov · 2023-03-27T09:05:37Z

Codecov Report

Patch coverage: 83.80% and project coverage change: +0.09 🎉

Comparison is base (8c0456b) 49.48% compared to head (095cb58) 49.57%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #13018      +/-   ##
==========================================
+ Coverage   49.48%   49.57%   +0.09%     
==========================================
  Files         256      256              
  Lines       43844    43920      +76     
==========================================
+ Hits        21695    21773      +78     
+ Misses      19988    19985       -3     
- Partials     2161     2162       +1

Impacted Files	Coverage Δ
cmd/argocd/commands/admin/cluster.go	`0.00% <0.00%> (ø)`
controller/appcontroller.go	`53.35% <0.00%> (-0.04%)`	⬇️
util/env/env.go	`96.07% <50.00%> (-3.93%)`	⬇️
controller/sharding/sharding.go	`93.06% <96.55%> (+44.58%)`	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

controller/sharding/sharding.go

akram · 2023-06-03T08:55:08Z

@leoluz gentle ping to re-enable auto-merge for this one. I amended last commit after updating the manifests to align them with the env variable ARGOCD_CONTROLLER_SHARDING_ALGORITHM.

…pe of functions to filter clusters by shard - Adding unit tests for sharding - Refresh clusters list on DistributionFunction call Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: ishitasequeira <ishiseq29@gmail.com>

…it size from [strconv.Atoi](1) to a lower bit size type uint32 without an upper bound check. Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Signed-off-by: Raghavi Shirur <rshirur@redhat.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

…e sharding algo env var Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

…ontroller.sharding.algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

…bution methods name, ran codegen on manifests Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

bygui86 · 2023-06-05T14:04:32Z

@leoluz amazing that this feature got merged! In which version is going to be released and when?
thanks to all guys!!

leoluz · 2023-06-05T14:45:24Z

@bygui86 In the Projects section of this PR (right panel) you can see the roadmap it is associated to and the release day planned for that. In case you can't see it, it is Argo CD 2.8 (~Aug 07th 2023)

bygui86 · 2023-06-05T14:46:32Z

@leoluz great thanks now I see it!

bofeng96 · 2023-06-27T16:51:55Z

Appreciated!

bofeng96 · 2023-06-27T17:02:38Z

I'm not sure if other users have this requirement. I believe it's better to let one application controller shard handle in-cluster only, while balancing other clusters across other application controller shards.

bygui86 · 2023-06-27T20:06:27Z

@bofeng96 may I ask you why do you have such requirement?
what if you have hundreds or even thousands ArgoCD App for in-cluster? there you really sharding...

bofeng96 · 2023-06-27T22:35:25Z

@bofeng96 may I ask you why do you have such requirement? what if you have hundreds or even thousands ArgoCD App for in-cluster? there you really sharding...

We do have thousands of parent Apps on in-cluster, and that's the reason we want the one shard only deal with those apps. It's too heavy if shard 0 is still taking care of other clusters, as the workload on in-cluster is more heavier. The current homogeneous distribution algorithm is only used to assign same # of clusters on each shards.

bygui86 · 2023-06-28T06:35:53Z

@bofeng96 you can still use the not-so-well-documented manually assignment...
As described here, you can set the "shard" in the Cluster K8s Secret. The only requirement is that if you configure N shards, then you must have exactly N application controller replicas.

apiVersion: v1
kind: Secret
metadata:
  name: my-cluster
type: Opaque
data:
  config: base64config
  name: base64name
  server: base64server
  shard: base64shard

anandf · 2023-04-05T13:41:44Z

cmd/argocd/commands/admin/cluster.go

-				clusterShard = sharding.GetShardByID(cluster.ID, replicas)
+				distributionFunction := sharding.GetDistributionFunction(kubeClient, settingsMgr)
+				distributionFunction(&cluster)
+				cluster.Shard = pointer.Int64Ptr(int64(clusterShard))


cluster.Shard should not be updated. That field is meant for users to explicitly set static shard value so that right application controller instance with that shard value picks it up.

anandf · 2023-04-05T13:43:57Z

controller/appcontroller.go

-			ctrl.projByNameCache.Delete(key)
-			return true
-		})
+		if ctrl != nil {


Is this nil check really needed ?

anandf · 2023-04-05T13:47:46Z

controller/sharding/sharding.go

+
+	return func(c *v1alpha1.Cluster) int {
+		if c != nil && replicas != 0 {
+			clusterIndex, ok := clusterIndexdByClusterId[c.ID]


clusterIndexdByClusterId is initialized only during the GetShardByIndexModuloReplicasCountDistributionFunction creation. Any clusters added post that initialization would not be available in the map. This map should be dynamically created within the filter function.

anandf · 2023-04-08T12:36:45Z

controller/sharding/sharding.go

+	case filterFunctionName == "hash":
+		distributionFunction = GetShardByIndexModuloReplicasCountDistributionFunction(db)
+	case filterFunctionName == "legacy":
+	default:


The default behaviour should be legacy, So can't we just have 2 cases case filterFunctionName == "hash" and default ? Also it would be better if this logic could be moved to cmd/argocd-application-controller/controllers/argocd_application_control.go where it would be possible to switch the algorithm using command line flags.

anandf · 2023-04-08T12:40:50Z

controller/sharding/sharding.go

+	return clusters
+}
+
+func createClusterIndexByClusterIdMap(db db.ArgoDB) map[string]int {


Can we add some comments for the newly added functions ?

anandf · 2023-05-18T16:08:25Z

cmd/argocd/commands/admin/cluster.go

 			if replicas > 0 {
-				clusterShard = sharding.GetShardByID(cluster.ID, replicas)
+				distributionFunction := sharding.GetDistributionFunction(argoDB, common.DefaultShardingAlgorithm)
+				distributionFunction(&cluster)


Should this line be clusterShard=distributionFunction(&cluster) ?

bofeng96 · 2023-06-28T08:53:50Z

@bofeng96 you can still use the not-so-well-documented manually assignment... As described here, you can set the "shard" in the Cluster K8s Secret. The only requirement is that if you configure N shards, then you must have exactly N application controller replicas.
apiVersion: v1
kind: Secret
metadata:
  name: my-cluster
type: Opaque
data:
  config: base64config
  name: base64name
  server: base64server
  shard: base64shard

haha. That's the workaround which I've been using. The sweet problem is that I have to edit new Secrets monthly because the # of clusters we managed is increasing.

…ernative algorithms (argoproj#13018) * fix: Extraction of DistributionFunction to allow passing different type of functions to filter clusters by shard - Adding unit tests for sharding - Refresh clusters list on DistributionFunction call Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: ishitasequeira <ishiseq29@gmail.com> * fix: Incorrect conversion of an integer with architecture-dependent bit size from [strconv.Atoi](1) to a lower bit size type uint32 without an upper bound check. Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Added config to switch to round-robin sharding Signed-off-by: Raghavi Shirur <rshirur@redhat.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Documenting sharding more, adding shuffling tests (skipped), re-enable sharding algo env var Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Allow configuration through argocd-cmd-params-cm configMap and key: controller.sharding.algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * De-duplicate code, remove reflection for default case, shorten distribution methods name, ran codegen on manifests Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> --------- Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: ishitasequeira <ishiseq29@gmail.com> Signed-off-by: Raghavi Shirur <rshirur@redhat.com> Co-authored-by: Raghavi Shirur <raghaviss11@gmail.com>

akram force-pushed the fix-round-robin-cluster-sharding branch from e04a107 to 1da356c Compare March 27, 2023 07:46

akram changed the title ~~Use db.clusterList index as the sharding placement parameter~~ fix: Use db.clusterList index as the sharding placement parameter Mar 27, 2023

akram force-pushed the fix-round-robin-cluster-sharding branch 2 times, most recently from dda4713 to 8e67a27 Compare March 27, 2023 08:52

akram force-pushed the fix-round-robin-cluster-sharding branch 4 times, most recently from 17b3986 to d4f8ef3 Compare March 29, 2023 10:00

raghavi101 mentioned this pull request Mar 30, 2023

chore(cli): Added config to switch to round-robin sharding akram/argo-cd#6

Merged

akram marked this pull request as draft April 3, 2023 12:41

akram force-pushed the fix-round-robin-cluster-sharding branch from d4f8ef3 to 17b3986 Compare April 3, 2023 16:10

akram mentioned this pull request Apr 3, 2023

WIP: Dynamic function filter #13087

Closed

13 tasks

akram force-pushed the fix-round-robin-cluster-sharding branch 2 times, most recently from e7b258b to 24ef5ea Compare April 3, 2023 18:34

akram changed the title ~~fix: Use db.clusterList index as the sharding placement parameter~~ fix: Sort cluster uids and use them as a sharding placement parameter Apr 3, 2023

akram marked this pull request as ready for review April 4, 2023 08:26

akram marked this pull request as draft April 4, 2023 08:28

akram force-pushed the fix-round-robin-cluster-sharding branch 7 times, most recently from eb005bf to a2afda7 Compare April 4, 2023 08:58

github-advanced-security bot found potential problems Apr 4, 2023

View reviewed changes

controller/sharding/sharding.go Fixed Show fixed Hide fixed

akram force-pushed the fix-round-robin-cluster-sharding branch 4 times, most recently from f079c26 to 39429e6 Compare April 4, 2023 12:37

akram force-pushed the fix-round-robin-cluster-sharding branch 3 times, most recently from 7457a40 to d07922a Compare June 1, 2023 14:35

akram force-pushed the fix-round-robin-cluster-sharding branch 2 times, most recently from 77916ea to be58bca Compare June 4, 2023 12:42

akram and others added 6 commits June 5, 2023 06:31

fix: Incorrect conversion of an integer with architecture-dependent b…

19f19d8

…it size from [strconv.Atoi](1) to a lower bit size type uint32 without an upper bound check. Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Added config to switch to round-robin sharding

d4c18c8

Signed-off-by: Raghavi Shirur <rshirur@redhat.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Documenting sharding more, adding shuffling tests (skipped), re-enabl…

d395186

…e sharding algo env var Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Allow configuration through argocd-cmd-params-cm configMap and key: c…

c7ea155

…ontroller.sharding.algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

De-duplicate code, remove reflection for default case, shorten distri…

095cb58

…bution methods name, ran codegen on manifests Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

akram force-pushed the fix-round-robin-cluster-sharding branch from be58bca to 095cb58 Compare June 5, 2023 04:31

leoluz merged commit ee983fe into argoproj:master Jun 5, 2023
24 checks passed

anandf reviewed Jun 28, 2023

View reviewed changes

agaudreault mentioned this pull request Jul 4, 2023

Sharding algorithm constantly evaluated waste CPU and create too many logs #14337

Closed

3 tasks

This was referenced Jul 28, 2023

REQUEST: Promotion to Approver(docs) for @todaywasawesome argoproj/argoproj#228

Closed

Load between controllers (argocd-application-controller) is not evenly distributed #6125

Open

phanama mentioned this pull request Nov 22, 2023

fix(cli): pass redis compression to cluster stats and shards commands (#16060) #16421

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Fixes sharding placement algorithm and allows development of alternative algorithms #13018

fix: Fixes sharding placement algorithm and allows development of alternative algorithms #13018

akram commented Mar 27, 2023 •

edited

codecov bot commented Mar 27, 2023 •

edited

akram commented Jun 3, 2023

bygui86 commented Jun 5, 2023

leoluz commented Jun 5, 2023

bygui86 commented Jun 5, 2023

bofeng96 commented Jun 27, 2023

bofeng96 commented Jun 27, 2023

bygui86 commented Jun 27, 2023

bofeng96 commented Jun 27, 2023

bygui86 commented Jun 28, 2023

anandf Apr 5, 2023

anandf Apr 5, 2023

anandf Apr 5, 2023

anandf Apr 8, 2023

anandf Apr 8, 2023

anandf May 18, 2023

bofeng96 commented Jun 28, 2023

fix: Fixes sharding placement algorithm and allows development of alternative algorithms #13018

fix: Fixes sharding placement algorithm and allows development of alternative algorithms #13018

Conversation

akram commented Mar 27, 2023 • edited

codecov bot commented Mar 27, 2023 • edited

Codecov Report

akram commented Jun 3, 2023

bygui86 commented Jun 5, 2023

leoluz commented Jun 5, 2023

bygui86 commented Jun 5, 2023

bofeng96 commented Jun 27, 2023

bofeng96 commented Jun 27, 2023

bygui86 commented Jun 27, 2023

bofeng96 commented Jun 27, 2023

bygui86 commented Jun 28, 2023

anandf Apr 5, 2023

Choose a reason for hiding this comment

anandf Apr 5, 2023

Choose a reason for hiding this comment

anandf Apr 5, 2023

Choose a reason for hiding this comment

anandf Apr 8, 2023

Choose a reason for hiding this comment

anandf Apr 8, 2023

Choose a reason for hiding this comment

anandf May 18, 2023

Choose a reason for hiding this comment

bofeng96 commented Jun 28, 2023

akram commented Mar 27, 2023 •

edited

codecov bot commented Mar 27, 2023 •

edited