Skip to content

Commit

Permalink
MB-52109 planner - tenant aware rebalance
Browse files Browse the repository at this point in the history
Implement sub-cluster/tenant affinity aware planner which moves tenants
from nodes above HWM usage threshold to nodes below LWM usage threshold.

The core algorithm works as follows:
1. Identify the nodes above HWM usage threshold.
2. Identify the candidate tenants to move out such that resource usage comes
   below LWM threshold after the rebalance.
3. Identify the target nodes below LWM usage threshold.
4. Place the candidate tenants on target nodes upto LWM usage
   threshold.

The following cases will be addressed in future commits:
1. Replica repair
2. Moving indexes from ejected nodes/swap rebalance.

Change-Id: I0ff1fed18b764595b6907f461b86bc11c2424ad2
  • Loading branch information
deepkaran committed Sep 14, 2022
1 parent 0bd80dd commit 9172585
Show file tree
Hide file tree
Showing 5 changed files with 707 additions and 5 deletions.
8 changes: 8 additions & 0 deletions secondary/common/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -3311,6 +3311,14 @@ var SystemConfig = Config{
false, // case-insensitive
},

"indexer.settings.units_quota": ConfigValue{
uint64(5000 * 20),
"Maximum RU/WU quota for indexer",
uint64(5000 * 20),
false, // mutable
false, // case-insensitive
},

"indexer.settings.serverless.indexLimit": ConfigValue{
200,
"Limit on the number of indexes that can be created per bucket in Serverless Mode.",
Expand Down
12 changes: 9 additions & 3 deletions secondary/indexer/rebalancer.go
Original file line number Diff line number Diff line change
Expand Up @@ -386,9 +386,15 @@ func (r *Rebalancer) initRebalAsync() {
}

start := time.Now()
r.transferTokens, hostToIndexToRemove, err = planner.ExecuteRebalance(cfg["clusterAddr"].String(), *r.topologyChange,
r.nodeUUID, onEjectOnly, disableReplicaRepair, threshold, timeout, cpuProfile,
minIterPerTemp, maxIterPerTemp)

if c.IsServerlessDeployment() {
r.transferTokens, hostToIndexToRemove, err = planner.ExecuteTenantAwareRebalance(cfg["clusterAddr"].String(),
*r.topologyChange, r.nodeUUID)
} else {
r.transferTokens, hostToIndexToRemove, err = planner.ExecuteRebalance(cfg["clusterAddr"].String(), *r.topologyChange,
r.nodeUUID, onEjectOnly, disableReplicaRepair, threshold, timeout, cpuProfile,
minIterPerTemp, maxIterPerTemp)
}
if err != nil {
l.Errorf("Rebalancer::initRebalAsync Planner Error %v", err)
go r.finishRebalance(err)
Expand Down
1 change: 1 addition & 0 deletions secondary/indexer/stats_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -832,6 +832,7 @@ func (s *IndexerStats) SetPlannerFilters() {
s.memoryUsedStorage.AddFilter(stats.PlannerFilter)
s.memoryUsed.AddFilter(stats.PlannerFilter)
s.memoryQuota.AddFilter(stats.PlannerFilter)
s.memoryRss.AddFilter(stats.PlannerFilter)
s.uptime.AddFilter(stats.PlannerFilter)
s.cpuUtilization.AddFilter(stats.PlannerFilter)
}
Expand Down
Loading

0 comments on commit 9172585

Please sign in to comment.