Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](routine-load) optimize allocate task to be algorithm for load balance #32021

Merged
merged 1 commit into from Mar 20, 2024

Conversation

HHoflittlefish777
Copy link
Contributor

@HHoflittlefish777 HHoflittlefish777 commented Mar 9, 2024

Proposed changes

The task always running on one BE may cause the BE to run too many tasks simultaneously while the other BE is not running few task. Here, a simple limitation is added to optimize this situation.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@HHoflittlefish777
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Mar 9, 2024

PR approved by anyone and no changes requested.

@HHoflittlefish777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38261 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72a0e9d98388da6dff99073a7bd49e63918bafaf, data reload: false

------ Round 1 ----------------------------------
q1	17624	4935	4136	4136
q2	2026	151	141	141
q3	10605	1089	916	916
q4	6741	757	703	703
q5	7467	2613	2619	2613
q6	183	123	126	123
q7	1253	821	800	800
q8	9321	1979	2012	1979
q9	7129	6420	6422	6420
q10	8510	3490	3616	3490
q11	423	226	222	222
q12	709	295	305	295
q13	17788	2865	2867	2865
q14	277	243	254	243
q15	499	454	451	451
q16	517	399	397	397
q17	948	563	554	554
q18	7190	6518	6346	6346
q19	4921	1417	1385	1385
q20	532	292	284	284
q21	6497	3613	3581	3581
q22	361	342	317	317
Total cold run time: 111521 ms
Total hot run time: 38261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4171	4114	4114	4114
q2	321	231	222	222
q3	2996	2829	2845	2829
q4	1874	1550	1602	1550
q5	5250	5263	5240	5240
q6	195	117	121	117
q7	2221	1832	1817	1817
q8	3152	3312	3308	3308
q9	8573	8526	8551	8526
q10	3710	3679	3656	3656
q11	534	458	448	448
q12	715	547	555	547
q13	16922	2881	2857	2857
q14	276	268	249	249
q15	481	460	445	445
q16	444	401	431	401
q17	1751	1505	1461	1461
q18	7670	7176	7018	7018
q19	1637	1514	1568	1514
q20	1938	1714	1721	1714
q21	4843	4567	4794	4567
q22	530	462	460	460
Total cold run time: 70204 ms
Total hot run time: 53060 ms

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 16, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@liaoxin01
Copy link
Contributor

run p0

@liaoxin01
Copy link
Contributor

run performance

@liaoxin01 liaoxin01 merged commit e7d2b38 into apache:master Mar 20, 2024
28 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants