Cell as SKU in intra-vc scheduler #34

abuccts · 2021-01-28T11:18:07Z

New cell as SKU feature in intra-vc scheduler, work items:

new interface for pod group and scheduling result placement
pod request parsing and conversion
affinity group / pod group adjustments before intra-vc scheduler
new scheduling algorithm in intra-vc scheduler
scheduling placements adjustments
update original test cases (leaf cell) Convert old test cases to v2 schema #37
spec, request examples and document, design document
add new test cases (within feature) Add new test cases for V2 schema #38
end-to-end tests and debug

Add new interface for pod group, including scheduling request, interal status, and scheduled placement.

Add conversion, defaulting, and validation for v2 pod scheduling spec, update ExtractPodSchedulingSpec accordingly.

Convert AlgoAffinityGroup to PodGroupSchedulingStatus before intra VC scheduler.

Add cell type to level mapping for intra-vc.

Fix pod group placement type in scheduling status.

* adjust virtual and physical placements for scheduling results * update pod binding info for tree structure * expose pod group in api through webserver

* pick best affinity cells * convert cells to leaf cells in placement

Fix legacy unit tests.

codecov-commenter · 2021-04-20T13:01:04Z

Codecov Report

Merging #34 (5c1e5c8) into master (66f26ac) will decrease coverage by 7.78%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
- Coverage   88.46%   80.68%   -7.79%     
==========================================
  Files           8       10       +2     
  Lines        1890     2205     +315     
==========================================
+ Hits         1672     1779     +107     
- Misses        166      334     +168     
- Partials       52       92      +40

Flag	Coverage Δ
unittests	`80.68% <ø> (-7.79%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ft/hivedscheduler/pkg/algorithm/cell_allocation.go	`59.90% <0.00%> (-32.37%)`	⬇️
...om/microsoft/hivedscheduler/pkg/algorithm/utils.go	`76.09% <0.00%> (-9.30%)`	⬇️
...ft/hivedscheduler/pkg/algorithm/hived_algorithm.go	`76.59% <0.00%> (-8.65%)`	⬇️
...hivedscheduler/pkg/algorithm/intra_vc_scheduler.go	`100.00% <0.00%> (ø)`
...cheduler/pkg/algorithm/topology_aware_scheduler.go
...microsoft/hivedscheduler/pkg/algorithm/types_v2.go	`89.82% <0.00%> (ø)`
...dscheduler/pkg/algorithm/hived_algorithm_tester.go	`66.42% <0.00%> (ø)`
...uler/pkg/algorithm/topology_guarantee_scheduler.go	`92.88% <0.00%> (ø)`
...m/microsoft/hivedscheduler/pkg/algorithm/config.go	`96.46% <0.00%> (+0.11%)`	⬆️
...om/microsoft/hivedscheduler/pkg/algorithm/types.go	`90.62% <0.00%> (+5.58%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66f26ac...5c1e5c8. Read the comment docs.

pkg/algorithm/sku_scheduler.go

pkg/algorithm/hived_algorithm_v2_test.go

pkg/algorithm/sku_scheduler.go

Fix bugs in new v2 test cases, including: * calculate leaf cell numbers per pod when creating new PodGroupSchedulingStatus * get cell chain through within cell type (no higher than node) instead of leaf cell type * sort cells by virtual address in cluster view

Fix comments.

pkg/api/v2/types.go

yqwang-ms · 2021-09-14T04:15:12Z

pkg/api/v2/types.go

+
+// PodGroupMemberCellSpec represents cell spec for each pod in pod group.
+type PodGroupMemberCellSpec struct {
+	CellType   api.CellType `yaml:"cellType"`   // cell type to be used in pod, differnt group can have differnt cell typs in the same chain, TODO: not support multiple chains yet


typs

types #Pending

yqwang-ms · 2021-09-14T04:27:52Z

pkg/api/v2/types.go

+	Name          string               `yaml:"name"`          // pod group name
+	WithinOneCell api.CellType         `yaml:"withinOneCell"` // within cell for all cells in current group, e.g., two GPU-cell within one numa-cell, two node-cell within one rack-cell
+	Pods          []PodGroupMemberSpec `yaml:"pods"`          // pod list for current group, any two pods are gang with each other in the groupS
+	ChildGroups   []*PodGroupSpec      `yaml:"childGroups"`   // child group in the hierarchical structure


Pods and ChildGroups are order sensitive? #Pending

yqwang-ms · 2021-09-14T09:18:35Z

Only backtrace to parent pod group may not enough to find the whole pod root group placement if it exist in theory.
Consider the following example:

Only backtrace to parent, result in no solution, but in theory, a solution exist.

We can add more pod group sorting criterias, such as if it has withinOneCell requirement, but still cannot prove only backtrace to parent is enough.

So given pod group trees are generally small trees with few nodes, backtrace to previous bound pod group (i.e. parent or left sibling node) should not too time consuming, which is enough to find the whole pod root group placement if it exist in theory.

In reply to: 918971150

yqwang-ms · 2021-09-14T11:20:35Z

Add demo here, such as pods withinOneCell and single pod's non-leaf cell type #WontFix

Refers to: example/feature/README.md:2 in c5e1bf1. [](commit_id = c5e1bf1, deletion_comment = False)

yqwang-ms · 2021-09-14T11:28:48Z

pkg/algorithm/sku_scheduler.go

+// virtual cluster view
+type skuClusterView []*skuCell
+
+// skuScheduler can schedule a pod group of pods with arbitrary cell types on a cluster view.


with arbitrary cell types

Also mentioned the withinOneCell feature?
Such as can be:
skuScheduler can schedule a group of pods with guaranteed affinity requirement (withinOneCell, e.g. with in one rack) and each pod can specify arbitrary cell types (e.g. non-leaf cell). #Pending

yqwang-ms · 2021-09-14T11:31:33Z

pkg/algorithm/sku_scheduler.go

+// skuScheduler can schedule a pod group of pods with arbitrary cell types on a cluster view.
+// It first tries to place pod group without preemption, then enable preemption if schedule failed.
+// For each try, it will find within cells for each pod group, then find cells for each pods with better affinity.
+type skuScheduler struct {


skuScheduler

Why name it as skuScheduler? topologyGuaranteeScheduler is better? #Pending

yqwang-ms · 2021-09-14T11:34:40Z

pkg/algorithm/hived_algorithm.go

 	cellChains map[string][]CellChain
 	// map each level in a chain to the specific cell type name
 	cellTypes map[CellChain]map[CellLevel]api.CellType
+	// map each type name in a chain to the specific cell level


type name

cell type #Pending

yqwang-ms · 2021-09-14T13:15:40Z

pkg/algorithm/sku_scheduler.go

+	placement, failedReason = PodGroupPlacement{}, "no matched cells in vc"
+
+	cv := skuClusterView{nil}
+	if level, ok := s.cellLevels[podGroup.WithinOneCell]; ok {


WithinOneCell

If no WithinOneCell defined in cluster, early fail? #Pending

yqwang-ms · 2021-09-14T13:32:43Z

pkg/algorithm/sku_scheduler.go

+
+func (s *skuScheduler) findCellsForPods(
+	pods []apiv2.PodGroupMemberSpec,
+	priority CellPriority,


early return if pods nil or empty? #Pending

yqwang-ms · 2021-09-14T13:37:50Z

pkg/algorithm/sku_scheduler.go

+	placement, failedReason = PodGroupPlacement{}, "no matched cells in vc"
+
+	cv := skuClusterView{nil}
+	if level, ok := s.cellLevels[podGroup.WithinOneCell]; ok {


WithinOneCell

What if root group WithinOneCell is not specified? Such as from v1 #Closed

yqwang-ms · 2021-09-15T02:50:08Z

pkg/algorithm/sku_scheduler.go

+}
+
+func (s *skuScheduler) createSkuClusterView(
+	within *skuCell,


Comment the func return and its param relation #Pending

yqwang-ms · 2021-09-15T02:51:54Z

pkg/algorithm/sku_scheduler.go

+}
+
+func (s *skuScheduler) createSkuClusterView(
+	within *skuCell,


within

withinCell? #Pending

yqwang-ms · 2021-09-15T02:57:38Z

pkg/algorithm/sku_scheduler.go

+
+// skuCell type for selected level cell in virtual cluster view
+type skuCell struct {
+	cell                          Cell            // within cell level, maybe higher or lower than node level


within cell level

within cell instead of just a level? #Pending

yqwang-ms · 2021-09-15T03:22:38Z

pkg/algorithm/sku_scheduler.go

+	return false
+}
+
+func ancestorNoHigherThanLevel(withinLevel CellLevel, cell Cell) Cell {


ancestorNoHigherThanLevel

ancestorNoLowerThanLevel? #Pending

yqwang-ms · 2021-09-15T03:27:23Z

pkg/algorithm/sku_scheduler.go

+		for _, c := range s.chainCellList[l] {
+			if (within != nil && !isAncestor(within.cell, c)) ||
+				cv.contains(ancestorNoHigherThanLevel(withinLevel, c)) {
+				continue


Will "cv.contains() continue" cause some cells with more free gpus are excluded in the final cv? #Closed

yqwang-ms · 2021-09-15T04:22:06Z

pkg/algorithm/sku_scheduler.go

+				"have to use at least one bad cell %v", withinCell.address)
+		}
+		podPlacement := s.findCellsForSinglePod(pods[podIndex], priority, withinCell, allocatedCells)
+		if podPlacement == nil {


schedule pods also need backtrace to previous bound pod? otherwise, we need to prove it will not miss solution #Pending

since we have changed podGroup.Pods to podGroup.Pod, all pods here with different podIndex are the same now

yqwang-ms · 2021-09-15T04:26:42Z

pkg/algorithm/sku_scheduler.go

+func (s *skuScheduler) findCellsForSinglePod(
+	pod apiv2.PodGroupMemberSpec,
+	priority CellPriority,
+	within *skuCell,


within

withinCell? #Pending

yqwang-ms · 2021-09-15T06:44:09Z

pkg/algorithm/sku_scheduler.go

+		}
+		podPlacement := s.findCellsForSinglePod(pods[podIndex], priority, withinCell, allocatedCells)
+		if podPlacement == nil {
+			withinCellIndex++


withinCellIndex

Do we need to reset withinCellIndex = 0 before schedule each pod? #Pending

since we have changed podGroup.Pods to podGroup.Pod, all pods here with different podIndex are the same, no need to reset withinCellIndex anymore

yqwang-ms · 2021-09-15T06:54:58Z

pkg/algorithm/sku_scheduler.go

+	freeCells, preemptibleCells = getFreeCellsAtLevel(
+		within.cell, currLevel, priority, allocatedCells, freeCells, preemptibleCells)
+	// free leaf cells will be used first (before preemptible leaf cells)
+	freeCells = append(freeCells, preemptibleCells...)


freeCells

availableCells? #Pending

yqwang-ms · 2021-09-15T07:00:05Z

pkg/algorithm/sku_scheduler.go

+}
+
+func (s *skuScheduler) findCellsForSinglePod(
+	pod apiv2.PodGroupMemberSpec,


Comment for single pod we still best effort find nearest cells for it (old behaiour) #Pending

yqwang-ms · 2021-09-15T07:11:48Z

pkg/algorithm/sku_scheduler.go

+	currLevel := s.cellLevels[pod.CellsPerPod.CellType]
+	freeCells, preemptibleCells := CellList{}, CellList{}
+	freeCells, preemptibleCells = getFreeCellsAtLevel(
+		within.cell, currLevel, priority, allocatedCells, freeCells, preemptibleCells)


First use cell that does not need to be got from higher level cell split? #Pending

sort cells when splitting in getFreeCellsAtLevel so that cells need higher level split can be used later

yqwang-ms · 2021-09-16T05:26:50Z

pkg/algorithm/hived_algorithm.go

 	// map each leaf cell type to all chains that contain this type
+	leafCellChains map[string][]CellChain
+	// map each within cell type to all chains that contain this type
 	cellChains map[string][]CellChain
 	// map each level in a chain to the specific cell type name


type name

cell type #Pending

yqwang-ms · 2021-09-16T06:12:13Z

pkg/algorithm/sku_scheduler.go

+// 1. cell health (prefer healthy)
+// 2. cell level (prefer lower)
+// 3. usedLeafCellNumAtPriority (prefer higher)
+// 4. usedLeafCellNumHigherPriority (prefer lower)


if greedy search, may need use same order to sort here? #Pending

Comment usedLeafCellNumAtPriority different meaning in different crossPriorityPack case

yqwang-ms · 2021-09-17T06:17:29Z

pkg/api/v2/types.go

+	Name          string               `yaml:"name"`          // pod group name
+	WithinOneCell api.CellType         `yaml:"withinOneCell"` // within cell for all cells in current group, e.g., two GPU-cell within one numa-cell, two node-cell within one rack-cell
+	Pods          []PodGroupMemberSpec `yaml:"pods"`          // pod list for current group, any two pods are gang with each other in the groupS
+	ChildGroups   []*PodGroupSpec      `yaml:"childGroups"`   // child group in the hierarchical structure


Seems no need Pod list, and then pods search no need back trace #Pending

changed to PodGroup.Pod, reserve Pods field to flatten Pod for internal structure

Update according to comments.

Fix GitHub Action config.

Change `PodGroup.Pods` to `PodGroup.Pod` to resolve backtracing search issue.

* early return check for pods and withinOneCell * sortcells when splitting in getFreeCellsAtLevel * add comments for sorting usedLeafCellNumAtPriority

Update config in examples accordingly.

Skip build for legacy topology aware scheduler through build tag.

abuccts added 2 commits January 28, 2021 15:46

Add new interface for pod group

b8e332b

Add new interface for pod group, including scheduling request, interal status, and scheduled placement.

Add conversion, defauling, validation for v2 spec

76fd537

Add conversion, defaulting, and validation for v2 pod scheduling spec, update ExtractPodSchedulingSpec accordingly.

abuccts added the enhancement New feature or request label Jan 28, 2021

abuccts requested review from hzhua and yqwang-ms January 28, 2021 11:18

abuccts self-assigned this Jan 28, 2021

abuccts mentioned this pull request Jan 28, 2021

v0.4.0 release plan #33

Open

9 tasks

abuccts requested a review from hzy46 February 3, 2021 06:31

Convert affinity group to pod group

e56fe3b

Convert AlgoAffinityGroup to PodGroupSchedulingStatus before intra VC scheduler.

abuccts force-pushed the xiongyf/cell-as-sku branch from db3d1ec to e56fe3b Compare March 4, 2021 02:52

abuccts and others added 6 commits March 12, 2021 16:11

Add cell type to level mapping for intra-vc

3b69b82

Add cell type to level mapping for intra-vc.

Fix pod group placement type in scheduling status

c509e4c

Fix pod group placement type in scheduling status.

Convert old test cases to v2 schema (#37)

9ce58b4

Update placements, binding info, and webserver

d041610

* adjust virtual and physical placements for scheduling results * update pod binding info for tree structure * expose pod group in api through webserver

Update intra vc scheduler

7d1275d

* pick best affinity cells * convert cells to leaf cells in placement

Fix legacy unit tests

3a7d752

Fix legacy unit tests.

hzhua reviewed May 10, 2021

View reviewed changes

pkg/algorithm/sku_scheduler.go Outdated Show resolved Hide resolved

hzhua reviewed May 10, 2021

View reviewed changes

pkg/algorithm/sku_scheduler.go Outdated Show resolved Hide resolved

hzhua reviewed May 10, 2021

View reviewed changes

pkg/algorithm/sku_scheduler.go Outdated Show resolved Hide resolved

hzhua reviewed May 11, 2021

View reviewed changes

pkg/algorithm/hived_algorithm_v2_test.go Show resolved Hide resolved

hzhua reviewed May 11, 2021

View reviewed changes

pkg/algorithm/sku_scheduler.go Outdated Show resolved Hide resolved

hzhua reviewed May 11, 2021

View reviewed changes

pkg/algorithm/sku_scheduler.go Outdated Show resolved Hide resolved

abuccts force-pushed the xiongyf/cell-as-sku branch from 9fd0560 to ae26632 Compare May 12, 2021 09:02

abuccts marked this pull request as ready for review May 12, 2021 09:06

Fix comments

0f86361

Fix comments.

yqwang-ms reviewed Jun 21, 2021

View reviewed changes

pkg/api/v2/types.go Outdated Show resolved Hide resolved

yqwang-ms reviewed Jun 21, 2021

View reviewed changes

pkg/api/v2/types.go Outdated Show resolved Hide resolved

yqwang-ms reviewed Jun 21, 2021

View reviewed changes

pkg/api/v2/types.go Outdated Show resolved Hide resolved

yqwang-ms reviewed Sep 14, 2021

View reviewed changes

yqwang-ms reviewed Sep 15, 2021

View reviewed changes

yqwang-ms reviewed Sep 16, 2021

View reviewed changes

yqwang-ms reviewed Sep 17, 2021

View reviewed changes

abuccts added 6 commits November 8, 2021 16:21

Update according to comments

ed7577c

Update according to comments.

Fix GitHub Action config

35949b6

Fix GitHub Action config.

Change PodGroup.Pods to PodGroup.Pod

e71a5f6

Change `PodGroup.Pods` to `PodGroup.Pod` to resolve backtracing search issue.

Add early return, sort cells, and sorting comments

8a7f52d

* early return check for pods and withinOneCell * sortcells when splitting in getFreeCellsAtLevel * add comments for sorting usedLeafCellNumAtPriority

Update config in examples accordingly

cdaeea3

Update config in examples accordingly.

Skip build for legacy topo aware scheduler.

5c1e5c8

Skip build for legacy topology aware scheduler through build tag.

Cell as SKU in intra-vc scheduler #34

Are you sure you want to change the base?

Cell as SKU in intra-vc scheduler #34

Conversation

abuccts commented Jan 28, 2021 • edited

codecov-commenter commented Apr 20, 2021 • edited

Codecov Report

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms commented Sep 14, 2021 • edited

yqwang-ms commented Sep 14, 2021 • edited

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 14, 2021 • edited

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 15, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 16, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 16, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yqwang-ms Sep 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuccts commented Jan 28, 2021 •

edited

codecov-commenter commented Apr 20, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms commented Sep 14, 2021 •

edited

yqwang-ms commented Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 14, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 15, 2021 •

edited

yqwang-ms Sep 16, 2021 •

edited

yqwang-ms Sep 16, 2021 •

edited

yqwang-ms Sep 17, 2021 •

edited