[chore](load) adjust load_task_high_priority_threshold_second default value #50478

sollhui · 2025-04-27T13:41:11Z

What problem does this PR solve?

High priority load task was introduced to avoid load timeout with short execution times, which mainly for routine load tasks, now the timeout time for routine load tasks has been is set to 10 times the max_match_interval (#40818), so the default value of load_task_high_priority_threshold_second should also be increased.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-27T13:41:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2025-04-27T13:41:20Z

run buildall

liaoxin01

LGTM

github-actions · 2025-04-27T13:48:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-27T13:48:40Z

PR approved by anyone and no changes requested.

doris-robot · 2025-04-27T14:24:43Z

TPC-H: Total hot run time: 34851 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f85e2e9b77a4e002768163f83dbc45970bf9d73e, data reload: false

------ Round 1 ----------------------------------
q1	25905	5106	5048	5048
q2	2070	278	187	187
q3	10382	1245	690	690
q4	10227	1006	515	515
q5	7502	2369	2380	2369
q6	182	162	133	133
q7	904	762	609	609
q8	9322	1311	1133	1133
q9	6822	5204	5147	5147
q10	6867	2317	1925	1925
q11	501	285	267	267
q12	346	345	213	213
q13	17771	3735	3047	3047
q14	235	221	217	217
q15	537	485	482	482
q16	455	469	400	400
q17	598	865	373	373
q18	7690	7102	7060	7060
q19	1596	963	583	583
q20	338	333	224	224
q21	4162	3402	3290	3290
q22	1037	991	939	939
Total cold run time: 115449 ms
Total hot run time: 34851 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5210	5144	5204	5144
q2	237	346	228	228
q3	2184	2665	2284	2284
q4	1374	1816	1499	1499
q5	4495	4539	4478	4478
q6	229	172	134	134
q7	2045	1980	1759	1759
q8	2631	2748	2559	2559
q9	7332	7261	7113	7113
q10	2993	3181	2761	2761
q11	584	500	491	491
q12	661	738	612	612
q13	3491	3899	3345	3345
q14	312	300	276	276
q15	520	471	475	471
q16	461	502	469	469
q17	1157	1592	1379	1379
q18	7746	7717	7399	7399
q19	820	848	908	848
q20	1998	2020	1884	1884
q21	5228	5008	4728	4728
q22	1080	1085	1043	1043
Total cold run time: 52788 ms
Total hot run time: 50904 ms

doris-robot · 2025-04-27T14:36:38Z

TPC-DS: Total hot run time: 193349 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f85e2e9b77a4e002768163f83dbc45970bf9d73e, data reload: false

query1	1420	1085	1044	1044
query2	6148	1875	1864	1864
query3	11140	4647	4723	4647
query4	25752	24047	23336	23336
query5	4766	667	485	485
query6	302	200	193	193
query7	3983	489	280	280
query8	293	270	231	231
query9	8500	2576	2597	2576
query10	496	321	272	272
query11	15178	15108	14910	14910
query12	159	110	107	107
query13	1544	509	389	389
query14	9128	6106	6147	6106
query15	203	186	176	176
query16	7279	683	446	446
query17	1133	738	598	598
query18	1987	414	313	313
query19	202	186	167	167
query20	123	121	121	121
query21	216	123	108	108
query22	4507	4623	4338	4338
query23	34456	33673	33442	33442
query24	8582	2530	2472	2472
query25	518	471	401	401
query26	1167	272	155	155
query27	2942	513	339	339
query28	4675	2153	2154	2153
query29	743	557	458	458
query30	267	217	194	194
query31	914	841	804	804
query32	79	70	69	69
query33	579	388	337	337
query34	821	880	544	544
query35	808	821	764	764
query36	990	1037	888	888
query37	114	103	83	83
query38	4188	4288	4189	4189
query39	1513	1420	1469	1420
query40	219	129	113	113
query41	62	60	56	56
query42	121	111	112	111
query43	533	512	500	500
query44	1357	833	845	833
query45	198	184	217	184
query46	862	1036	638	638
query47	1858	1881	1861	1861
query48	391	419	309	309
query49	779	506	414	414
query50	662	700	426	426
query51	4313	4259	4323	4259
query52	110	116	113	113
query53	231	268	186	186
query54	598	592	501	501
query55	84	83	83	83
query56	308	295	293	293
query57	1187	1187	1147	1147
query58	274	271	281	271
query59	2817	2860	2793	2793
query60	337	319	308	308
query61	133	125	121	121
query62	774	741	696	696
query63	228	194	192	192
query64	4106	1027	697	697
query65	4441	4374	4436	4374
query66	1068	422	303	303
query67	16196	15549	15271	15271
query68	8223	890	519	519
query69	473	310	263	263
query70	1156	1074	1078	1074
query71	465	312	304	304
query72	5781	4784	4995	4784
query73	709	651	348	348
query74	8956	9122	8908	8908
query75	3924	3248	2743	2743
query76	3660	1202	760	760
query77	798	366	289	289
query78	10043	10094	9395	9395
query79	2270	806	560	560
query80	580	527	486	486
query81	496	260	222	222
query82	454	120	96	96
query83	246	247	220	220
query84	241	102	92	92
query85	790	366	305	305
query86	388	315	306	306
query87	4352	4441	4291	4291
query88	3615	2232	2218	2218
query89	407	318	282	282
query90	1864	218	211	211
query91	137	139	119	119
query92	79	62	55	55
query93	1933	946	586	586
query94	661	411	275	275
query95	374	294	291	291
query96	499	561	275	275
query97	3172	3210	3100	3100
query98	227	230	205	205
query99	1605	1400	1295	1295
Total cold run time: 280640 ms
Total hot run time: 193349 ms

doris-robot · 2025-04-27T14:41:52Z

ClickBench: Total hot run time: 29.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f85e2e9b77a4e002768163f83dbc45970bf9d73e, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.25	0.20	0.20
query4	1.60	0.19	0.11
query5	0.57	0.54	0.57
query6	1.21	0.72	0.73
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.59	0.58	0.58
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.61	0.60
query14	1.19	1.20	1.22
query15	0.89	0.87	0.86
query16	0.38	0.40	0.40
query17	1.02	1.03	1.04
query18	0.22	0.19	0.20
query19	1.92	1.77	1.85
query20	0.02	0.02	0.01
query21	15.37	0.94	0.55
query22	0.76	1.20	0.97
query23	14.69	1.37	0.63
query24	7.17	0.66	1.08
query25	0.51	0.28	0.07
query26	0.61	0.16	0.14
query27	0.06	0.06	0.04
query28	9.82	0.86	0.43
query29	12.57	3.94	3.24
query30	0.27	0.09	0.07
query31	2.82	0.59	0.39
query32	3.22	0.54	0.47
query33	3.03	2.98	3.03
query34	15.61	5.13	4.50
query35	4.59	4.56	4.52
query36	0.67	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.84 s
Total hot run time: 29.56 s

doris-robot · 2025-04-27T15:22:07Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.38% (14741/27107)
Line Coverage	43.46% (128956/296745)
Region Coverage	42.16% (65867/156218)
Branch Coverage	36.68% (33177/90450)

sollhui · 2025-04-27T15:52:47Z

run buildall

doris-robot · 2025-04-27T16:23:29Z

TPC-H: Total hot run time: 35104 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b45254e632a859989dfecd145e18aa61614b16b, data reload: false

------ Round 1 ----------------------------------
q1	26372	5312	5347	5312
q2	2114	293	179	179
q3	10374	1382	717	717
q4	10225	1083	547	547
q5	7575	2591	2722	2591
q6	201	176	136	136
q7	1011	831	646	646
q8	9323	1477	1207	1207
q9	6859	5256	5295	5256
q10	6908	2373	1921	1921
q11	490	308	267	267
q12	365	396	214	214
q13	17771	3889	3136	3136
q14	241	243	229	229
q15	553	491	505	491
q16	464	472	414	414
q17	623	890	378	378
q18	7576	7154	7225	7154
q19	1272	1086	577	577
q20	329	345	219	219
q21	4123	3675	2546	2546
q22	1036	1048	967	967
Total cold run time: 115805 ms
Total hot run time: 35104 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5504	5483	5491	5483
q2	249	345	234	234
q3	2230	2764	2412	2412
q4	1521	2005	1480	1480
q5	4731	4489	4444	4444
q6	257	187	129	129
q7	2119	1964	1827	1827
q8	2901	2964	2870	2870
q9	7229	7035	7133	7035
q10	3129	3341	2842	2842
q11	626	521	503	503
q12	699	793	590	590
q13	3628	4141	3296	3296
q14	287	305	280	280
q15	548	480	490	480
q16	500	540	476	476
q17	1277	1739	1461	1461
q18	7847	7547	7374	7374
q19	823	956	1252	956
q20	2004	2083	1909	1909
q21	5457	4938	4839	4839
q22	1109	1080	1046	1046
Total cold run time: 54675 ms
Total hot run time: 51966 ms

doris-robot · 2025-04-27T16:35:28Z

TPC-DS: Total hot run time: 193253 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b45254e632a859989dfecd145e18aa61614b16b, data reload: false

query1	1401	1078	1052	1052
query2	6184	1845	1858	1845
query3	11058	4557	4519	4519
query4	26238	24388	23756	23756
query5	4852	642	440	440
query6	305	195	197	195
query7	3990	513	280	280
query8	293	241	236	236
query9	8518	2531	2557	2531
query10	487	328	289	289
query11	15465	15149	14963	14963
query12	154	114	104	104
query13	1553	499	396	396
query14	9665	6073	6117	6073
query15	217	189	166	166
query16	7329	643	485	485
query17	1114	752	596	596
query18	2030	416	331	331
query19	194	177	161	161
query20	133	125	121	121
query21	201	163	111	111
query22	4552	4620	4399	4399
query23	34688	33689	33328	33328
query24	7976	2382	2377	2377
query25	538	501	441	441
query26	1183	279	170	170
query27	2907	505	346	346
query28	4973	2130	2139	2130
query29	740	590	497	497
query30	293	234	198	198
query31	927	879	803	803
query32	72	62	66	62
query33	551	391	323	323
query34	808	859	527	527
query35	823	826	754	754
query36	976	1009	902	902
query37	120	102	76	76
query38	4308	4235	4249	4235
query39	1536	1502	1486	1486
query40	211	118	108	108
query41	53	52	51	51
query42	118	104	109	104
query43	523	509	494	494
query44	1308	823	812	812
query45	179	181	164	164
query46	848	1018	631	631
query47	1892	1899	1833	1833
query48	384	445	301	301
query49	787	512	425	425
query50	638	705	402	402
query51	4311	4342	4215	4215
query52	103	103	95	95
query53	227	252	183	183
query54	570	619	510	510
query55	83	86	75	75
query56	319	303	312	303
query57	1241	1270	1146	1146
query58	262	259	256	256
query59	2751	2783	2749	2749
query60	340	310	307	307
query61	130	127	130	127
query62	766	770	667	667
query63	221	186	183	183
query64	4059	1043	695	695
query65	4570	4393	4450	4393
query66	1036	410	318	318
query67	15832	15324	15296	15296
query68	8448	889	510	510
query69	462	308	264	264
query70	1204	1161	1078	1078
query71	528	316	297	297
query72	5760	4786	4780	4780
query73	684	591	337	337
query74	8929	9104	8673	8673
query75	3937	3232	2700	2700
query76	3754	1211	755	755
query77	789	370	278	278
query78	10608	10467	9477	9477
query79	2862	898	550	550
query80	693	504	436	436
query81	497	260	222	222
query82	648	122	93	93
query83	283	255	237	237
query84	296	100	81	81
query85	783	348	304	304
query86	375	306	284	284
query87	4435	4401	4394	4394
query88	3507	2177	2172	2172
query89	411	315	276	276
query90	1875	204	208	204
query91	146	147	117	117
query92	73	70	53	53
query93	1937	964	575	575
query94	654	409	314	314
query95	367	283	300	283
query96	486	561	268	268
query97	3226	3225	3192	3192
query98	226	208	203	203
query99	1487	1429	1299	1299
Total cold run time: 283484 ms
Total hot run time: 193253 ms

doris-robot · 2025-04-27T16:40:39Z

ClickBench: Total hot run time: 29.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b45254e632a859989dfecd145e18aa61614b16b, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.26	0.20	0.20
query4	1.59	0.20	0.20
query5	0.60	0.57	0.59
query6	1.16	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.53	0.51
query10	0.56	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	1.19	1.16	1.21
query15	0.85	0.85	0.84
query16	0.39	0.39	0.39
query17	1.03	1.01	1.00
query18	0.21	0.20	0.19
query19	1.94	1.79	1.79
query20	0.01	0.01	0.02
query21	15.39	0.89	0.56
query22	0.76	1.26	0.70
query23	14.84	1.34	0.63
query24	6.84	1.42	0.64
query25	0.50	0.07	0.12
query26	0.71	0.16	0.13
query27	0.05	0.06	0.05
query28	9.48	0.91	0.44
query29	12.54	4.01	3.38
query30	0.26	0.09	0.08
query31	2.81	0.60	0.38
query32	3.23	0.54	0.46
query33	3.03	3.08	3.00
query34	15.75	5.12	4.48
query35	4.48	4.50	4.51
query36	0.67	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.33 s
Total hot run time: 29.44 s

hello-stephen · 2025-04-27T17:32:12Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	54.39% (14744/27107)
Line Coverage	43.48% (129033/296740)
Region Coverage	42.19% (65895/156201)
Branch Coverage	36.71% (33194/90432)

hello-stephen · 2025-04-27T18:36:27Z

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.63% (14803/26611)
Line Coverage	45.25% (134064/296256)
Region Coverage	42.20% (77051/182603)
Branch Coverage	36.16% (37240/102982)

dataroaring

LGTM

hello-stephen · 2025-04-28T04:04:13Z

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.45% (14755/26611)
Line Coverage	45.04% (133444/296256)
Region Coverage	41.97% (76647/182603)
Branch Coverage	35.95% (37025/102982)

… value (#50478) High priority load task was introduced to avoid load timeout with short execution times, which mainly for routine load tasks, now the timeout time for routine load tasks has been is set to 10 times the max_match_interval (#40818), so the default value of load_task_high_priority_threshold_second should also be increased.

…cond default value #50478 (#50522) Cherry-picked from #50478 Co-authored-by: hui lai <laihui@selectdb.com>

… value (apache#50478) High priority load task was introduced to avoid load timeout with short execution times, which mainly for routine load tasks, now the timeout time for routine load tasks has been is set to 10 times the max_match_interval (apache#40818), so the default value of load_task_high_priority_threshold_second should also be increased.

liaoxin01 added dev/2.1.x dev/3.0.x labels Apr 27, 2025

liaoxin01 approved these changes Apr 27, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2025

github-actions bot added the reviewed label Apr 27, 2025

sollhui force-pushed the high_prio_flush branch 3 times, most recently from ccc62e3 to d2f1c1f Compare April 27, 2025 15:52

adjust load_task_high_priority_threshold_second default value

1b45254

sollhui force-pushed the high_prio_flush branch from d2f1c1f to 1b45254 Compare April 27, 2025 15:52

dataroaring approved these changes Apr 28, 2025

View reviewed changes

liaoxin01 merged commit 7b88230 into apache:master Apr 28, 2025
23 of 24 checks passed

github-actions bot mentioned this pull request Apr 28, 2025

branch-3.0: [chore](load) adjust load_task_high_priority_threshold_second default value #50478 #50522

Merged

github-actions bot mentioned this pull request Apr 28, 2025

branch-2.1: [chore](load) adjust load_task_high_priority_threshold_second default value #50478 #50523

Closed

dataroaring pushed a commit that referenced this pull request Apr 29, 2025

branch-3.0: [chore](load) adjust load_task_high_priority_threshold_se…

a14458c

…cond default value #50478 (#50522) Cherry-picked from #50478 Co-authored-by: hui lai <laihui@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels Apr 29, 2025

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Open

[chore](load) adjust load_task_high_priority_threshold_second default value #50478

[chore](load) adjust load_task_high_priority_threshold_second default value #50478

Uh oh!

Conversation

sollhui commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 27, 2025

Uh oh!

sollhui commented Apr 27, 2025

Uh oh!

liaoxin01 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 27, 2025

Uh oh!

github-actions bot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

BE UT Coverage Report

Uh oh!

sollhui commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

doris-robot commented Apr 27, 2025

Uh oh!

hello-stephen commented Apr 27, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 27, 2025

BE Regression P0 && UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Apr 28, 2025

BE Regression P0 && UT Coverage Report

Uh oh!

Uh oh!

Uh oh!

sollhui commented Apr 27, 2025 •

edited

Loading