feat: Consistent hashing with bounded loads algorithm for sharding #16564

akram · 2023-12-07T00:09:33Z

Checklist:

codecov · 2023-12-07T00:18:40Z

Codecov Report

Attention: Patch coverage is 51.06383% with 92 lines in your changes missing coverage. Please review.

Project coverage is 44.92%. Comparing base (69e82cc) to head (427bd5a).
Report is 1 commits behind head on master.

Files	Patch %	Lines
controller/sharding/consistent/consistent.go	38.46%	75 Missing and 5 partials ⚠️
controller/sharding/sharding.go	83.63%	6 Missing and 3 partials ⚠️
cmd/argocd/commands/admin/cluster.go	0.00%	2 Missing ⚠️
...ntroller/commands/argocd_application_controller.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #16564      +/-   ##
==========================================
+ Coverage   44.90%   44.92%   +0.02%     
==========================================
  Files         354      355       +1     
  Lines       47705    47889     +184     
==========================================
+ Hits        21421    21516      +95     
- Misses      23488    23569      +81     
- Partials     2796     2804       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

akram · 2023-12-07T07:32:30Z

Enhancement proposal is here #16570 and it has been exposed briefly during several calls, but, we will discuss more deeply

controller/sharding/consistent/consistent.go

controller/sharding/sharding.go

ishitasequeira

@akram Last couple of things (mainly documentation), otherwise the PR looks good to merge!!

I think we missed adding "consistent-hashing" as a new option in cli commands in below places:

argo-cd/cmd/argocd/commands/admin/cluster.go

Line 222 in f16b339

    
           command.Flags().StringVar(&shardingAlgorithm, "sharding-method", common.DefaultShardingAlgorithm, "Sharding method. Defaults: legacy. Supported sharding methods are : [legacy, round-robin] ")

argo-cd/cmd/argocd/commands/admin/cluster.go

Line 517 in f16b339

    
           command.Flags().StringVar(&shardingAlgorithm, "sharding-method", common.DefaultShardingAlgorithm, "Sharding method. Defaults: legacy. Supported sharding methods are : [legacy, round-robin] ")

argo-cd/cmd/argocd-application-controller/commands/argocd_application_controller.go

Line 223 in f16b339

    
           command.Flags().StringVar(&shardingAlgorithm, "sharding-method", env.StringFromEnv(common.EnvControllerShardingAlgorithm, common.DefaultShardingAlgorithm), "Enables choice of sharding method. Supported sharding methods are : [legacy, round-robin] ")

Also, could you add some documentation regarding the feature in https://github.com/ishitasequeira/argo-cd/blob/f16b339bbfd17d93bdfe38c8f05cb7f4b4844123/docs/operator-manual/high_availability.md

akram · 2024-05-30T15:11:01Z

@akram Last couple of things (mainly documentation), otherwise the PR looks good to merge!!

I think we missed adding "consistent-hashing" as a new option in cli commands in below places:

argo-cd/cmd/argocd/commands/admin/cluster.go

Line 222 in f16b339

command.Flags().StringVar(&shardingAlgorithm, "sharding-method", common.DefaultShardingAlgorithm, "Sharding method. Defaults: legacy. Supported sharding methods are : [legacy, round-robin] ")

done ✅

argo-cd/cmd/argocd/commands/admin/cluster.go

Line 517 in f16b339

command.Flags().StringVar(&shardingAlgorithm, "sharding-method", common.DefaultShardingAlgorithm, "Sharding method. Defaults: legacy. Supported sharding methods are : [legacy, round-robin] ")

done ✅

argo-cd/cmd/argocd-application-controller/commands/argocd_application_controller.go

Line 223 in f16b339

command.Flags().StringVar(&shardingAlgorithm, "sharding-method", env.StringFromEnv(common.EnvControllerShardingAlgorithm, common.DefaultShardingAlgorithm), "Enables choice of sharding method. Supported sharding methods are : [legacy, round-robin] ")

done ✅

Also, could you add some documentation regarding the feature in https://github.com/ishitasequeira/argo-cd/blob/f16b339bbfd17d93bdfe38c8f05cb7f4b4844123/docs/operator-manual/high_availability.md

and done ✅

ishitasequeira

@akram there seems to be issues with the generated code. You would need to run codegen

controller/sharding/sharding.go

Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

- The assignment or running of the algorithm has to be consistent across all the clusters. Changed the function to return a map where the consistent hash will be used to build the map - Modifications to the createConsistentHashsingWithBoundLoads function. This will create the map for cluster to shard. Note that the list must be consistent across all shards so that is why the cluster list must be sorted before going through the consistent hash algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

ishitasequeira

Thanks @akram !! LGTM!!

…rgoproj#16564) * Adds consistent hashing with bound loads sharding algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Make the assignement consistent accross all clusters - The assignment or running of the algorithm has to be consistent across all the clusters. Changed the function to return a map where the consistent hash will be used to build the map - Modifications to the createConsistentHashsingWithBoundLoads function. This will create the map for cluster to shard. Note that the list must be consistent across all shards so that is why the cluster list must be sorted before going through the consistent hash algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Extracting constant and simplifying boolean expression Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Update docs: consistent-hashing sharding algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> --------- Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> Signed-off-by: rumstead <37445536+rumstead@users.noreply.github.com>

…rgoproj#16564) * Adds consistent hashing with bound loads sharding algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Make the assignement consistent accross all clusters - The assignment or running of the algorithm has to be consistent across all the clusters. Changed the function to return a map where the consistent hash will be used to build the map - Modifications to the createConsistentHashsingWithBoundLoads function. This will create the map for cluster to shard. Note that the list must be consistent across all shards so that is why the cluster list must be sorted before going through the consistent hash algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Extracting constant and simplifying boolean expression Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> * Update docs: consistent-hashing sharding algorithm Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com> --------- Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

akram requested review from a team as code owners December 7, 2023 00:09

akram changed the title ~~Consistent sharding with bounded loads~~ feat: Consistent sharding with bounded loads Dec 7, 2023

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from 629cf2a to a2714e0 Compare December 7, 2023 00:59

akram marked this pull request as draft December 7, 2023 07:50

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from a2714e0 to 5350c9c Compare December 11, 2023 13:46

akram marked this pull request as ready for review December 11, 2023 13:48

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 8 times, most recently from 9d86b5c to 08e7298 Compare January 17, 2024 14:17

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 4 times, most recently from 5cf46ad to dfc7af1 Compare January 25, 2024 15:50

Enclavet mentioned this pull request Jan 27, 2024

feat: Add app data to sharding cache to allow sharding by apps #17014

Merged

14 tasks

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 2 times, most recently from e2d5bf7 to d143e9f Compare January 29, 2024 16:00

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 2 times, most recently from 9b2ad1f to 20581cf Compare February 7, 2024 04:41

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from 20581cf to ca5fadf Compare February 26, 2024 06:18

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from ca5fadf to d20f4f3 Compare March 4, 2024 16:24

Enclavet reviewed Mar 11, 2024

View reviewed changes

controller/sharding/consistent/consistent.go Outdated Show resolved Hide resolved

Enclavet reviewed Mar 11, 2024

View reviewed changes

controller/sharding/sharding.go Outdated Show resolved Hide resolved

todaywasawesome approved these changes May 17, 2024

View reviewed changes

ishitasequeira requested changes May 20, 2024

View reviewed changes

akram changed the title ~~feat: Consistent sharding with bounded loads~~ feat: Consistent hashing with bounded loads algorithm for sharding May 21, 2024

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 2 times, most recently from e5ae056 to bb60944 Compare May 22, 2024 08:36

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 3 times, most recently from aba8794 to c7d5f4f Compare May 30, 2024 15:09

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from c7d5f4f to 41840e8 Compare June 3, 2024 09:47

ishitasequeira reviewed Jun 3, 2024

View reviewed changes

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from 41840e8 to e617637 Compare June 3, 2024 14:28

ishitasequeira reviewed Jun 3, 2024

View reviewed changes

controller/sharding/sharding.go Outdated Show resolved Hide resolved

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch 2 times, most recently from 0e0d564 to 2e5c3a9 Compare June 4, 2024 13:45

akram added 4 commits June 4, 2024 17:21

Adds consistent hashing with bound loads sharding algorithm

ca2e87a

Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Extracting constant and simplifying boolean expression

0cf03cd

Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

Update docs: consistent-hashing sharding algorithm

b349f81

Signed-off-by: Akram Ben Aissi <akram.benaissi@gmail.com>

akram force-pushed the consistent-sharding-with-bounded-loads-akram branch from 2e5c3a9 to b349f81 Compare June 4, 2024 16:21

ishitasequeira approved these changes Jun 5, 2024

View reviewed changes

ishitasequeira enabled auto-merge (squash) June 5, 2024 02:30

Merge branch 'master' into consistent-sharding-with-bounded-loads-akram

7dd101d

Enclavet mentioned this pull request Jun 5, 2024

REQUEST: New membership for Enclavet argoproj/argoproj#302

Open

Merge branch 'master' into consistent-sharding-with-bounded-loads-akram

427bd5a

ishitasequeira merged commit 75cd97d into argoproj:master Jun 5, 2024
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Consistent hashing with bounded loads algorithm for sharding #16564

feat: Consistent hashing with bounded loads algorithm for sharding #16564

akram commented Dec 7, 2023 •

edited

codecov bot commented Dec 7, 2023 •

edited

akram commented Dec 7, 2023

ishitasequeira left a comment

akram commented May 30, 2024 •

edited

ishitasequeira left a comment

ishitasequeira left a comment

feat: Consistent hashing with bounded loads algorithm for sharding #16564

feat: Consistent hashing with bounded loads algorithm for sharding #16564

Conversation

akram commented Dec 7, 2023 • edited

codecov bot commented Dec 7, 2023 • edited

Codecov Report

akram commented Dec 7, 2023

ishitasequeira left a comment

Choose a reason for hiding this comment

akram commented May 30, 2024 • edited

ishitasequeira left a comment

Choose a reason for hiding this comment

ishitasequeira left a comment

Choose a reason for hiding this comment

akram commented Dec 7, 2023 •

edited

codecov bot commented Dec 7, 2023 •

edited

akram commented May 30, 2024 •

edited