disttask: subtasks rebalance during task execution #48306

ywqzzy · 2023-11-06T04:04:38Z

What problem does this PR solve?

Issue Number: ref #48795 close #47298

Problem Summary:
See issue #47298

What is changed and how it works?

RebalanceSubtasks process flow:

find out nodes need to clean subtasks.
get subtasks for each node before scaling out.
group subtasks for each scheduler.
prepare subtasks that need to rebalance to other nodes.
rebalance subtasks to other nodes.
rebalance rest subtasks evenly to liveNodes.
update subtasks and do clean up logic.

Behavior:
see ut in rebalance_test.go

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

first run on 1 tidb then scale out 1 tidb.

from 2 tidb to 1 tidb

from 1 tidb to 3 tidb

from 3 tidb to 1 tidb

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2023-11-06T04:04:52Z

Hi @ywqzzy. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…ce_subtasks

codecov · 2023-11-06T10:43:53Z

Codecov Report

Merging #48306 (a9085c2) into master (333e8c3) will increase coverage by 1.5864%.
Report is 3 commits behind head on master.
The diff coverage is 0.0000%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #48306        +/-   ##
================================================
+ Coverage   70.9893%   72.5757%   +1.5864%     
================================================
  Files          1368       1391        +23     
  Lines        404075     412255      +8180     
================================================
+ Hits         286850     299197     +12347     
+ Misses        97242      94193      -3049     
+ Partials      19983      18865      -1118

Flag	Coverage Δ
integration	`43.9004% <0.0000%> (?)`
unit	`70.9929% <ø> (+0.0036%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9663% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.3702% <ø> (-4.6121%)`	⬇️

…ce_subtasks

pkg/disttask/framework/dispatcher/dispatcher.go

ywqzzy · 2023-11-29T07:19:58Z

/cc @tangenta

…ce_subtasks

Benjamin2037

LGTM

ti-chi-bot · 2023-11-29T15:31:57Z

[LGTM Timeline notifier]

Timeline:

2023-11-28 09:35:44.560812451 +0000 UTC m=+915373.226038645: ☑️ agreed by D3Hunter.
2023-11-29 15:31:55.598954008 +0000 UTC m=+1023144.264180204: ☑️ agreed by Benjamin2037.

tangenta

rest lgtm

tangenta · 2023-11-29T15:37:17Z

pkg/disttask/framework/dispatcher/dispatcher.go

+		// When no tidb scale-in/out and averageSubtaskCnt*len(d.LiveNodes) < len(subtasks),
+		// no need to send subtask to other nodes.
+		// eg: tidb1 with 3 subtasks, tidb2 with 2 subtasks, subtasks are balanced now.
+		if averageSubtaskCnt*len(d.LiveNodes) < len(subtasks) && len(d.TaskNodes) == len(d.LiveNodes) {


averageSubtaskCnt*len(d.LiveNodes) < len(subtasks) seems to be always true?

No, eg:
tidb1: 4 subtasks, tidb2: 2 subtasks, then average = 3,
average * len(liveNodes) = 6 = len(subtasks)

…ce_subtasks

ywqzzy · 2023-11-30T02:43:37Z

/hold

ti-chi-bot · 2023-11-30T02:53:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, D3Hunter, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Benjamin2037,D3Hunter,tangenta]
~~pkg/ddl/OWNERS~~ [Benjamin2037,tangenta]
~~pkg/disttask/OWNERS~~ [Benjamin2037,D3Hunter,tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ywqzzy · 2023-11-30T02:54:37Z

/unhold

ti-chi-bot · 2023-11-30T04:27:38Z

In response to a cherrypick label: new pull request created to branch release-7.5: #49029.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

done

705286e

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 6, 2023

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label Nov 6, 2023

ywqzzy added 3 commits November 6, 2023 13:12

minor refine

6b3a574

Merge branch 'master' of https://github.com/pingcap/tidb into rebalan…

efac7c7

…ce_subtasks

scale in load balance, more tests

45b3e90

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 6, 2023

for test

e21d314

ywqzzy added 2 commits November 6, 2023 22:07

tmp save

2c6db0e

refine tests, fix update subtasks' bug

d9c59a9

ywqzzy changed the title ~~disttask: fix subtasks unbalanced when node restart~~ [WIP] disttask: fix subtasks unbalanced when node restart Nov 7, 2023

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2023

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 17, 2023

ywqzzy added 4 commits November 22, 2023 10:10

Merge branch 'master' of https://github.com/pingcap/tidb into rebalan…

ec3e00d

…ce_subtasks

fix

cdbefe0

fix build

56e209a

tmp save

17893de

ywqzzy changed the title ~~[WIP] disttask: fix subtasks unbalanced when node restart~~ [WIP] disttask: rebalance subtasks Nov 23, 2023

ywqzzy changed the title ~~[WIP] disttask: rebalance subtasks~~ [WIP] disttask: subtasks rebalance during task execution Nov 23, 2023

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 23, 2023

ywqzzy force-pushed the rebalance_subtasks branch from c314142 to 17893de Compare November 23, 2023 03:57

ywqzzy added 2 commits November 23, 2023 12:06

fix

bdc02ee

modify log

892f37b

ywqzzy changed the title ~~[WIP] disttask: subtasks rebalance during task execution~~ disttask: subtasks rebalance during task execution Nov 23, 2023

D3Hunter reviewed Nov 28, 2023

View reviewed changes

pkg/disttask/framework/dispatcher/dispatcher.go Outdated Show resolved Hide resolved

pkg/disttask/framework/dispatcher/dispatcher.go Show resolved Hide resolved

D3Hunter approved these changes Nov 28, 2023

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm label Nov 28, 2023

rename

da202bb

ti-chi-bot bot requested a review from tangenta November 29, 2023 07:20

ywqzzy added 3 commits November 29, 2023 18:02

Merge branch 'master' of https://github.com/pingcap/tidb into rebalan…

ed6d55c

…ce_subtasks

fixbuild

ee3fc34

update

7dd80ec

Benjamin2037 approved these changes Nov 29, 2023

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm labels Nov 29, 2023

tangenta reviewed Nov 29, 2023

View reviewed changes

Merge branch 'master' of https://github.com/pingcap/tidb into rebalan…

c474988

…ce_subtasks

ywqzzy force-pushed the rebalance_subtasks branch from a48de64 to c474988 Compare November 30, 2023 01:55

ywqzzy added 2 commits November 30, 2023 10:19

fix table bug

2c00680

fix testutil

a9085c2

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 30, 2023

tangenta approved these changes Nov 30, 2023

View reviewed changes

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 30, 2023

ti-chi-bot bot merged commit ef02d72 into pingcap:master Nov 30, 2023
15 checks passed

ti-chi-bot mentioned this pull request Nov 30, 2023

disttask: subtasks rebalance during task execution (#48306) #49029

Closed

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Nov 30, 2023

This is an automated cherry-pick of pingcap#48306

483a94a

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ywqzzy mentioned this pull request Jan 16, 2024

Enhance distributed task execution framework 2.0 #48795

Closed

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disttask: subtasks rebalance during task execution #48306

disttask: subtasks rebalance during task execution #48306

ywqzzy commented Nov 6, 2023 •

edited

tiprow bot commented Nov 6, 2023

codecov bot commented Nov 6, 2023 •

edited

ywqzzy commented Nov 29, 2023

Benjamin2037 left a comment

ti-chi-bot bot commented Nov 29, 2023

tangenta left a comment

tangenta Nov 29, 2023

ywqzzy Nov 30, 2023 •

edited

ywqzzy commented Nov 30, 2023

ti-chi-bot bot commented Nov 30, 2023

ywqzzy commented Nov 30, 2023

ti-chi-bot commented Nov 30, 2023

disttask: subtasks rebalance during task execution #48306

disttask: subtasks rebalance during task execution #48306

Conversation

ywqzzy commented Nov 6, 2023 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

tiprow bot commented Nov 6, 2023

codecov bot commented Nov 6, 2023 • edited

Codecov Report

ywqzzy commented Nov 29, 2023

Benjamin2037 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Nov 29, 2023

[LGTM Timeline notifier]

tangenta left a comment

Choose a reason for hiding this comment

tangenta Nov 29, 2023

Choose a reason for hiding this comment

ywqzzy Nov 30, 2023 • edited

Choose a reason for hiding this comment

ywqzzy commented Nov 30, 2023

ti-chi-bot bot commented Nov 30, 2023

ywqzzy commented Nov 30, 2023

ti-chi-bot commented Nov 30, 2023

ywqzzy commented Nov 6, 2023 •

edited

codecov bot commented Nov 6, 2023 •

edited

ywqzzy Nov 30, 2023 •

edited