Skip to content

branch-3.0: [feature](restore) introduce AgentBoundedBatchTask to manage concurrent restore tasks #50740#50843

Merged
dataroaring merged 1 commit intobranch-3.0from
auto-pick-50740-branch-3.0
May 13, 2025
Merged

branch-3.0: [feature](restore) introduce AgentBoundedBatchTask to manage concurrent restore tasks #50740#50843
dataroaring merged 1 commit intobranch-3.0from
auto-pick-50740-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 13, 2025

Cherry-picked from #50740

@w41ter
[feature](restore) introduce AgentBoundedBatchTask to manage concurre…
e72df72 
…nt restore tasks (#50740)

- Added a new class `AgentBoundedBatchTask` to limit the number of
concurrent tasks submitted to backends during restore operations.
- Updated `RestoreJob` to utilize `AgentBoundedBatchTask` instead of
`AgentBatchTask`, incorporating a new configuration parameter for
restore task concurrency.
- Introduced a new configuration field `restore_task_concurrency_per_be`
to specify the maximum number of concurrent restore tasks per backend.

Co-authored-by: wubiao02 <wubiao02@meituan.com>
@github-actions github-actions bot requested a review from dataroaring as a code owner May 13, 2025 07:12
@Thearas
Copy link
Contributor

Thearas commented May 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this May 13, 2025
@Thearas
Copy link
Contributor

Thearas commented May 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-H: Total hot run time: 40121 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e72df72fe5b1c52245172e0f6cab285fdff31ec1, data reload: false

------ Round 1 ----------------------------------
q1	17615	6990	6610	6610
q2	2047	171	164	164
q3	10619	1117	1143	1117
q4	10555	746	743	743
q5	7766	2914	2810	2810
q6	220	135	132	132
q7	996	615	602	602
q8	9359	1956	1980	1956
q9	6630	6360	6353	6353
q10	7053	2270	2288	2270
q11	471	263	265	263
q12	405	218	213	213
q13	17786	3082	3063	3063
q14	241	214	224	214
q15	511	471	481	471
q16	679	610	598	598
q17	1043	583	655	583
q18	7366	6670	6560	6560
q19	1376	1097	1089	1089
q20	484	203	201	201
q21	4007	3142	3249	3142
q22	1090	967	985	967
Total cold run time: 108319 ms
Total hot run time: 40121 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6634	6589	6586	6586
q2	329	231	234	231
q3	2916	2765	2882	2765
q4	2082	1828	1798	1798
q5	5833	5833	5907	5833
q6	222	136	133	133
q7	2254	1858	1880	1858
q8	3412	3650	3579	3579
q9	9032	8925	8988	8925
q10	3591	3501	3515	3501
q11	587	501	479	479
q12	791	582	597	582
q13	10172	3170	3166	3166
q14	293	266	255	255
q15	505	463	481	463
q16	695	638	649	638
q17	1924	1601	1601	1601
q18	8269	7886	7786	7786
q19	1710	1557	1426	1426
q20	2061	1849	1807	1807
q21	5570	5389	5330	5330
q22	1094	1092	1018	1018
Total cold run time: 69976 ms
Total hot run time: 59760 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

TPC-DS: Total hot run time: 198376 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e72df72fe5b1c52245172e0f6cab285fdff31ec1, data reload: false

query1	1285	916	914	914
query2	6256	2085	2043	2043
query3	10821	4271	4314	4271
query4	60735	34127	24094	24094
query5	5190	474	473	473
query6	336	193	174	174
query7	4933	315	323	315
query8	274	208	212	208
query9	6585	2589	2587	2587
query10	427	264	254	254
query11	16062	15317	15697	15317
query12	151	108	101	101
query13	1054	435	433	433
query14	10815	7462	6897	6897
query15	205	187	181	181
query16	7170	487	519	487
query17	1137	592	591	591
query18	1850	327	307	307
query19	204	165	166	165
query20	121	114	108	108
query21	211	106	106	106
query22	4788	4567	4704	4567
query23	34782	34042	34433	34042
query24	6359	2989	3044	2989
query25	523	425	428	425
query26	681	180	177	177
query27	2148	359	357	357
query28	4274	2456	2421	2421
query29	726	485	481	481
query30	242	157	156	156
query31	968	804	824	804
query32	62	57	54	54
query33	423	281	274	274
query34	908	503	527	503
query35	876	714	720	714
query36	1070	955	975	955
query37	118	71	69	69
query38	4098	3966	4005	3966
query39	1531	1471	1448	1448
query40	195	102	100	100
query41	51	48	49	48
query42	114	100	104	100
query43	543	514	491	491
query44	1177	818	814	814
query45	188	166	182	166
query46	1144	759	725	725
query47	2029	1877	1899	1877
query48	485	388	388	388
query49	721	411	400	400
query50	846	438	438	438
query51	7295	7367	7211	7211
query52	105	96	90	90
query53	264	185	184	184
query54	571	473	459	459
query55	77	75	85	75
query56	277	253	262	253
query57	1281	1136	1173	1136
query58	223	208	213	208
query59	3342	3087	3222	3087
query60	284	259	254	254
query61	144	108	106	106
query62	784	678	675	675
query63	221	188	194	188
query64	1631	685	629	629
query65	3300	3245	3273	3245
query66	706	294	298	294
query67	15989	15739	15600	15600
query68	3678	575	560	560
query69	423	262	262	262
query70	1161	1133	1101	1101
query71	335	280	257	257
query72	6378	4012	3994	3994
query73	764	345	354	345
query74	10095	9243	9100	9100
query75	3351	2633	2688	2633
query76	1736	1049	1096	1049
query77	490	272	275	272
query78	10631	9682	9681	9681
query79	1772	604	615	604
query80	887	441	449	441
query81	509	243	239	239
query82	1262	90	87	87
query83	243	146	146	146
query84	280	88	82	82
query85	895	301	292	292
query86	345	304	305	304
query87	4380	4263	4291	4263
query88	4001	2388	2379	2379
query89	417	288	300	288
query90	1978	189	186	186
query91	187	145	150	145
query92	61	49	49	49
query93	2158	567	559	559
query94	730	294	266	266
query95	356	260	259	259
query96	619	281	273	273
query97	3321	3110	3138	3110
query98	222	201	205	201
query99	1579	1280	1284	1280
Total cold run time: 309630 ms
Total hot run time: 198376 ms

@doris-robot
Copy link

doris-robot commented May 13, 2025

ClickBench: Total hot run time: 31.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e72df72fe5b1c52245172e0f6cab285fdff31ec1, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.11
query5	0.52	0.52	0.52
query6	1.14	0.73	0.72
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.49
query10	0.55	0.56	0.55
query11	0.13	0.12	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.73	2.71	2.75
query15	0.90	0.82	0.84
query16	0.37	0.38	0.37
query17	1.07	0.97	1.02
query18	0.23	0.21	0.21
query19	1.84	1.84	2.05
query20	0.01	0.01	0.01
query21	15.35	0.60	0.60
query22	2.50	2.70	1.49
query23	17.10	1.11	0.75
query24	3.37	1.36	1.01
query25	0.26	0.15	0.17
query26	0.45	0.15	0.13
query27	0.03	0.04	0.04
query28	9.80	0.55	0.45
query29	12.60	3.28	3.34
query30	0.25	0.08	0.07
query31	2.84	0.38	0.38
query32	3.26	0.48	0.46
query33	2.96	3.03	2.98
query34	17.10	4.49	4.44
query35	4.50	4.51	4.48
query36	0.69	0.47	0.51
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.4 s
Total hot run time: 31.77 s

dataroaring
dataroaring approved these changes May 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented May 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented May 13, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit f8a4636 into branch-3.0 May 13, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50740-branch-3.0 branch May 13, 2025 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Thearas @doris-robot @dataroaring @w41ter