proposal: Eviction Arbitration Mechanism in Descheduler #1454

baowj-678 · 2023-07-07T15:11:15Z

Ⅰ. Describe what this PR does

Arbitrate Mechanism is an important capability that Pod Migration relies on, and Pod Migration is relied on by many components (such as deschedulers). But pod Migration is a complex process, involving steps such as auditing, resource allocation, and application startup, and is mixed with application upgrading, scaling scenarios, and resource operation and maintenance operations by cluster administrators.

So when a large number of PODs are simultaneously migrated, this may have some impact on the stability of the system. In addition, if many pods of the same workload are migrated simultaneously, it will also have an impact on the stability of the application. Moreover, if a job's pod migration takes too long, it can affect the job's completion time.

Therefore, it is necessary to design an arbitration mechanism. This arbitration mechanism will select suitable PodMigrationJob to execute and control the execution speed of PodMigrationJob (to avoid a large number of jobs executing simultaneously), thereby ensuring the stability of the system and application.

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Design docs for #430

Ⅳ. Special notes for reviews

V. Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests
All checks passed in make test

codecov · 2023-07-07T15:21:54Z

Codecov Report

Patch coverage has no change and project coverage change: +1.02% 🎉

Comparison is base (06b2f5b) 64.14% compared to head (24ddf31) 65.16%.
Report is 62 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1454      +/-   ##
==========================================
+ Coverage   64.14%   65.16%   +1.02%     
==========================================
  Files         341      352      +11     
  Lines       34939    36326    +1387     
==========================================
+ Hits        22411    23673    +1262     
- Misses      10881    10909      +28     
- Partials     1647     1744      +97

Flag	Coverage Δ
unittests	`65.16% <ø> (+1.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 136 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: baowj-678 <bwj_678@qq.com>

docs/proposals/scheduling/20230707-eviction-arbitrate-mechanism-in-descheduler.md

1.update user stories 2.add diagrams 3.others Signed-off-by: baowj-678 <bwj_678@qq.com>

docs/proposals/scheduling/20230707-eviction-arbitrate-mechanism-in-descheduler.md

1.update diagrams. 2.move the arbitrate mechanism process before reconcile process. 3.delete the modifications to the PodMigrationJob CRD. 4.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra

Overall it looks good to me. But, it is recommended to add a picture to describe the timing diagram of a PMJ entering the migration controller, entering the arbitration queue, performing arbitration, and entering the workqueue. It will be easier to understand this way.

docs/proposals/scheduling/20230707-eviction-arbitrate-mechanism-in-descheduler.md

1.update diagrams. 2.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra · 2023-08-15T09:21:16Z

/lgtm
/approve

saintube

/lgtm

docs/proposals/scheduling/20230707-eviction-arbitrate-mechanism-in-descheduler.md

ZiMengSheng · 2023-08-20T07:27:01Z

/lgtm

jasonliu747

/lgtm

1.replace GroupFilter and Select process with Filter process. 2.update some diagrams. 3.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

1.add some introduction to the NonRetryableFilter and RetryableFilter. Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra

/lgtm

eahydra · 2023-08-22T11:24:43Z

@hormes PTAL

koordinator-bot · 2023-08-24T12:18:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eahydra, hormes, jasonliu747

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [hormes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…sh#1454) Signed-off-by: baowj-678 <bwj_678@qq.com>

koordinator-bot bot requested review from hormes and yihuifeng July 7, 2023 15:11

koordinator-bot bot added the size/L label Jul 7, 2023

[proposal] Eviction Arbitration Mechanism in Descheduler

6a2261a

Signed-off-by: baowj-678 <bwj_678@qq.com>

baowj-678 force-pushed the proposal_arbitration_mechanism branch from 81aa852 to 6a2261a Compare July 7, 2023 15:56

eahydra requested review from eahydra, FillZpp, ZiMengSheng, saintube, jasonliu747 and zwzhang0107 July 9, 2023 07:07

eahydra changed the title ~~[proposal] Eviction Arbitration Mechanism in Descheduler~~ proposal: Eviction Arbitration Mechanism in Descheduler Jul 9, 2023

eahydra reviewed Jul 19, 2023

View reviewed changes

Update proposal based on review comments:

ce6b0fd

1.update user stories 2.add diagrams 3.others Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra reviewed Aug 9, 2023

View reviewed changes

Update proposal based on review comments.

c3e095e

1.update diagrams. 2.move the arbitrate mechanism process before reconcile process. 3.delete the modifications to the PodMigrationJob CRD. 4.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra reviewed Aug 10, 2023

View reviewed changes

Update proposal based on review comments.

980c333

1.update diagrams. 2.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

koordinator-bot bot assigned eahydra Aug 15, 2023

koordinator-bot bot added the lgtm label Aug 15, 2023

saintube reviewed Aug 15, 2023

View reviewed changes

koordinator-bot bot assigned saintube Aug 15, 2023

ZiMengSheng reviewed Aug 16, 2023

View reviewed changes

docs/proposals/scheduling/20230707-eviction-arbitrate-mechanism-in-descheduler.md Outdated Show resolved Hide resolved

koordinator-bot bot assigned ZiMengSheng Aug 17, 2023

baowj-678 mentioned this pull request Aug 18, 2023

koord-descheduler: Arbitration Mechanism #1557

Closed

3 tasks

jasonliu747 approved these changes Aug 21, 2023

View reviewed changes

koordinator-bot bot assigned jasonliu747 Aug 21, 2023

Update proposal based on review comments.

fa9c364

1.replace GroupFilter and Select process with Filter process. 2.update some diagrams. 3.others. Signed-off-by: baowj-678 <bwj_678@qq.com>

koordinator-bot bot removed the lgtm label Aug 21, 2023

Update proposal based on review comments.

24ddf31

1.add some introduction to the NonRetryableFilter and RetryableFilter. Signed-off-by: baowj-678 <bwj_678@qq.com>

eahydra reviewed Aug 22, 2023

View reviewed changes

koordinator-bot bot added the lgtm label Aug 22, 2023

baowj-678 mentioned this pull request Aug 23, 2023

koord-descheduler: Add Arbitration Mechanism config #1571

Merged

3 tasks

hormes approved these changes Aug 24, 2023

View reviewed changes

koordinator-bot bot added the approved label Aug 24, 2023

koordinator-bot bot merged commit e06ce56 into koordinator-sh:main Aug 24, 2023
16 checks passed

baowj-678 mentioned this pull request Aug 26, 2023

koord-descheduler: add Arbitrator #1586

Merged

3 tasks

This was referenced Sep 7, 2023

koord-descheduler: add some SortFns for arbitration #1625

Merged

koord-descheduler: add arbitration to migration controller #1651

Merged

baowj-678 mentioned this pull request Nov 20, 2023

[REQUEST Membership]: New membership for baowj-678 koordinator-sh/community#5

Closed

6 tasks

ls-2018 pushed a commit to ls-2018/koordinator that referenced this pull request Mar 25, 2024

proposal: Eviction Arbitration Mechanism in Descheduler (koordinator-…

f4fe813

…sh#1454) Signed-off-by: baowj-678 <bwj_678@qq.com>

ls-2018 pushed a commit to ls-2018/koordinator that referenced this pull request Mar 25, 2024

proposal: Eviction Arbitration Mechanism in Descheduler (koordinator-…

4d1f839

…sh#1454) Signed-off-by: baowj-678 <bwj_678@qq.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proposal: Eviction Arbitration Mechanism in Descheduler #1454

proposal: Eviction Arbitration Mechanism in Descheduler #1454

baowj-678 commented Jul 7, 2023 •

edited by eahydra

codecov bot commented Jul 7, 2023 •

edited

eahydra left a comment

eahydra commented Aug 15, 2023

saintube left a comment

ZiMengSheng commented Aug 20, 2023

jasonliu747 left a comment

eahydra left a comment

eahydra commented Aug 22, 2023

koordinator-bot bot commented Aug 24, 2023

proposal: Eviction Arbitration Mechanism in Descheduler #1454

proposal: Eviction Arbitration Mechanism in Descheduler #1454

Conversation

baowj-678 commented Jul 7, 2023 • edited by eahydra

Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it

Ⅳ. Special notes for reviews

V. Checklist

codecov bot commented Jul 7, 2023 • edited

Codecov Report

eahydra left a comment

Choose a reason for hiding this comment

eahydra commented Aug 15, 2023

saintube left a comment

Choose a reason for hiding this comment

ZiMengSheng commented Aug 20, 2023

jasonliu747 left a comment

Choose a reason for hiding this comment

eahydra left a comment

Choose a reason for hiding this comment

eahydra commented Aug 22, 2023

koordinator-bot bot commented Aug 24, 2023

baowj-678 commented Jul 7, 2023 •

edited by eahydra

codecov bot commented Jul 7, 2023 •

edited