Skip to content

[enhance](load) support routine load job parallel begin and commit transaction #51893

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[enhance](load) support routine load job parallel begin and commit transaction #51893

wants to merge 1 commit into from

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Jun 18, 2025

What problem does this PR solve?

close #51892

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Jun 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34041 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95f061ac5208f16ed37e44546df530c7338f9eb5, data reload: false

------ Round 1 ----------------------------------
q1	17625	5218	4990	4990
q2	1923	286	184	184
q3	10327	1350	740	740
q4	10222	1009	516	516
q5	7579	2419	2386	2386
q6	185	163	130	130
q7	926	744	621	621
q8	9344	1298	1098	1098
q9	6773	5085	5082	5082
q10	6956	2421	1960	1960
q11	485	291	284	284
q12	341	340	213	213
q13	17772	3689	3063	3063
q14	225	232	208	208
q15	563	486	470	470
q16	433	429	368	368
q17	577	855	359	359
q18	7904	7270	7203	7203
q19	1210	965	560	560
q20	339	333	239	239
q21	3735	3150	2418	2418
q22	1027	999	949	949
Total cold run time: 106471 ms
Total hot run time: 34041 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5002	5060	5044	5044
q2	234	325	222	222
q3	2179	2648	2354	2354
q4	1334	1806	1339	1339
q5	4215	4161	4188	4161
q6	204	171	125	125
q7	1898	1835	1671	1671
q8	2521	2489	2444	2444
q9	6793	6680	6747	6680
q10	2950	3149	2731	2731
q11	564	516	481	481
q12	679	735	593	593
q13	3393	3697	3058	3058
q14	282	294	270	270
q15	525	466	466	466
q16	435	485	442	442
q17	1128	1470	1386	1386
q18	7243	7211	7057	7057
q19	771	751	817	751
q20	1921	1975	1822	1822
q21	4675	4401	4323	4323
q22	1049	1014	970	970
Total cold run time: 49995 ms
Total hot run time: 48390 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95f061ac5208f16ed37e44546df530c7338f9eb5, data reload: false

query1	840	401	383	383
query2	3187	1950	1955	1950
query3	5952	223	231	223
query4	25935	24007	23201	23201
query5	1292	640	472	472
query6	297	243	192	192
query7	4576	495	280	280
query8	252	224	219	219
query9	7285	2619	2653	2619
query10	447	348	279	279
query11	15236	15098	14813	14813
query12	155	110	107	107
query13	1632	529	414	414
query14	7751	6343	6293	6293
query15	210	200	194	194
query16	5608	645	488	488
query17	1231	699	581	581
query18	2067	397	296	296
query19	191	176	153	153
query20	120	112	118	112
query21	200	124	102	102
query22	4150	4157	4047	4047
query23	34143	33262	33122	33122
query24	8581	2410	2436	2410
query25	573	456	397	397
query26	1401	269	150	150
query27	2642	510	347	347
query28	4244	2141	2125	2125
query29	897	547	449	449
query30	327	219	188	188
query31	923	845	780	780
query32	74	64	74	64
query33	555	361	320	320
query34	795	844	538	538
query35	831	799	733	733
query36	946	983	899	899
query37	117	103	84	84
query38	4281	4237	4116	4116
query39	1496	1402	1404	1402
query40	214	127	116	116
query41	71	65	70	65
query42	128	110	113	110
query43	508	519	493	493
query44	1312	831	839	831
query45	176	185	172	172
query46	840	1033	642	642
query47	1772	1788	1729	1729
query48	395	441	322	322
query49	980	502	428	428
query50	644	683	420	420
query51	4174	4294	4118	4118
query52	113	108	100	100
query53	226	269	189	189
query54	621	584	521	521
query55	90	82	84	82
query56	328	322	309	309
query57	1174	1207	1114	1114
query58	309	260	253	253
query59	2677	2789	2715	2715
query60	328	322	310	310
query61	126	130	126	126
query62	812	724	655	655
query63	224	185	227	185
query64	3425	997	672	672
query65	4278	4208	4229	4208
query66	923	421	316	316
query67	15757	15685	15382	15382
query68	8960	895	524	524
query69	457	314	270	270
query70	1244	1108	1158	1108
query71	523	318	288	288
query72	5894	4968	5076	4968
query73	736	734	353	353
query74	9234	9118	9038	9038
query75	3837	3207	2805	2805
query76	3711	1198	757	757
query77	607	368	292	292
query78	10163	10117	9271	9271
query79	2861	811	585	585
query80	684	513	472	472
query81	481	257	218	218
query82	457	131	101	101
query83	341	249	238	238
query84	299	111	91	91
query85	770	352	319	319
query86	334	341	274	274
query87	4421	4534	4349	4349
query88	3246	2232	2244	2232
query89	425	324	278	278
query90	1875	205	206	205
query91	143	142	114	114
query92	81	63	59	59
query93	2638	935	578	578
query94	609	412	303	303
query95	365	295	277	277
query96	485	578	278	278
query97	2684	2759	2646	2646
query98	223	214	201	201
query99	1395	1409	1261	1261
Total cold run time: 264248 ms
Total hot run time: 186878 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95f061ac5208f16ed37e44546df530c7338f9eb5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.43	0.43	0.40
query6	1.17	0.67	0.69
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.51
query10	0.58	0.59	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.63	0.60	0.60
query14	0.82	0.81	0.81
query15	0.89	0.85	0.87
query16	0.39	0.39	0.38
query17	1.05	1.10	1.08
query18	0.22	0.22	0.21
query19	1.88	1.85	1.80
query20	0.01	0.01	0.01
query21	15.44	0.92	0.56
query22	0.75	1.26	0.82
query23	14.76	1.34	0.65
query24	7.42	0.78	0.45
query25	0.49	0.13	0.09
query26	0.57	0.17	0.14
query27	0.06	0.05	0.06
query28	9.29	0.90	0.44
query29	12.59	3.98	3.38
query30	0.24	0.09	0.06
query31	2.83	0.60	0.39
query32	3.24	0.55	0.47
query33	3.12	3.06	3.08
query34	16.06	5.35	4.77
query35	4.81	4.82	4.78
query36	0.68	0.51	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.15	0.14
query41	0.09	0.02	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.81 s
Total hot run time: 29.32 s

@sollhui sollhui force-pushed the parallel_begin_and_commit_txn branch from 95f061a to d96cd7a Compare June 18, 2025 14:21
@sollhui
Copy link
Contributor Author

sollhui commented Jun 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34044 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d96cd7a6ebafbbcf7dca75c6d592ac59a9419694, data reload: false

------ Round 1 ----------------------------------
q1	17582	5149	4974	4974
q2	1917	295	185	185
q3	10339	1332	739	739
q4	10237	1003	515	515
q5	7590	2411	2358	2358
q6	185	163	130	130
q7	917	757	602	602
q8	9341	1252	1102	1102
q9	6872	5150	5097	5097
q10	6892	2379	1981	1981
q11	484	296	283	283
q12	341	342	210	210
q13	17794	3656	3066	3066
q14	236	229	228	228
q15	568	494	480	480
q16	428	425	381	381
q17	610	873	385	385
q18	7777	7298	7214	7214
q19	1276	950	573	573
q20	344	344	220	220
q21	3829	3168	2358	2358
q22	1022	989	963	963
Total cold run time: 106581 ms
Total hot run time: 34044 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5063	5021	5031	5021
q2	235	328	221	221
q3	2159	2615	2297	2297
q4	1358	1786	1369	1369
q5	4199	4088	4402	4088
q6	210	168	129	129
q7	2059	1933	1758	1758
q8	2569	2592	2499	2499
q9	7092	7101	7121	7101
q10	3038	3293	2894	2894
q11	576	526	505	505
q12	719	751	606	606
q13	3561	3817	3218	3218
q14	271	301	301	301
q15	513	491	464	464
q16	459	479	447	447
q17	1137	1571	1339	1339
q18	7785	7613	7377	7377
q19	778	880	1005	880
q20	1971	2062	1901	1901
q21	5004	4546	4502	4502
q22	1056	984	991	984
Total cold run time: 51812 ms
Total hot run time: 49901 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185131 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d96cd7a6ebafbbcf7dca75c6d592ac59a9419694, data reload: false

query1	1003	390	391	390
query2	6560	1812	1781	1781
query3	6757	232	234	232
query4	26408	23654	22837	22837
query5	4376	641	454	454
query6	296	198	187	187
query7	4622	483	283	283
query8	269	222	213	213
query9	8634	2613	2649	2613
query10	470	329	271	271
query11	15336	15023	14851	14851
query12	157	106	114	106
query13	1662	544	414	414
query14	8614	6053	6247	6053
query15	204	193	169	169
query16	7158	623	483	483
query17	1052	728	596	596
query18	1979	423	333	333
query19	206	192	160	160
query20	121	128	122	122
query21	227	124	107	107
query22	4102	4151	4043	4043
query23	34142	33032	33166	33032
query24	8533	2447	2367	2367
query25	552	462	394	394
query26	1229	269	153	153
query27	2769	503	343	343
query28	4321	2121	2107	2107
query29	774	581	433	433
query30	283	220	234	220
query31	920	848	778	778
query32	71	67	64	64
query33	549	356	311	311
query34	794	845	513	513
query35	777	807	710	710
query36	929	990	887	887
query37	111	100	73	73
query38	4053	4155	4012	4012
query39	1476	1409	1392	1392
query40	210	118	109	109
query41	62	58	58	58
query42	125	106	117	106
query43	486	495	472	472
query44	1296	823	817	817
query45	179	175	165	165
query46	828	1015	632	632
query47	1722	1792	1724	1724
query48	394	428	318	318
query49	736	493	379	379
query50	630	675	422	422
query51	4102	4243	4155	4155
query52	115	108	97	97
query53	230	257	183	183
query54	576	570	500	500
query55	89	81	86	81
query56	293	308	291	291
query57	1202	1202	1129	1129
query58	272	259	243	243
query59	2555	2657	2572	2572
query60	325	309	334	309
query61	125	123	122	122
query62	805	711	652	652
query63	225	185	193	185
query64	4424	1041	671	671
query65	4276	4167	4187	4167
query66	1149	413	315	315
query67	15790	15617	15472	15472
query68	8198	894	537	537
query69	479	300	329	300
query70	1192	1086	1048	1048
query71	439	317	298	298
query72	5649	4877	4874	4874
query73	728	662	361	361
query74	9393	8807	8638	8638
query75	3777	3176	2692	2692
query76	3637	1190	749	749
query77	786	376	283	283
query78	10120	10139	9390	9390
query79	2219	813	571	571
query80	592	530	479	479
query81	523	256	229	229
query82	454	124	93	93
query83	247	247	225	225
query84	272	102	90	90
query85	798	345	308	308
query86	396	302	278	278
query87	4390	4461	4318	4318
query88	3519	2287	2288	2287
query89	385	316	287	287
query90	1823	217	212	212
query91	145	144	110	110
query92	79	61	62	61
query93	1648	965	597	597
query94	676	425	348	348
query95	373	298	290	290
query96	492	560	277	277
query97	2747	2807	2600	2600
query98	234	208	222	208
query99	1459	1423	1260	1260
Total cold run time: 273586 ms
Total hot run time: 185131 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d96cd7a6ebafbbcf7dca75c6d592ac59a9419694, data reload: false

query1	0.03	0.04	0.02
query2	0.08	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.43	0.44	0.42
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.51
query10	0.57	0.58	0.56
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	0.81	0.83	0.81
query15	0.90	0.87	0.88
query16	0.38	0.39	0.38
query17	1.05	1.05	1.06
query18	0.24	0.21	0.20
query19	1.98	1.95	1.86
query20	0.01	0.02	0.01
query21	15.39	0.90	0.56
query22	0.76	1.18	0.70
query23	14.89	1.40	0.64
query24	6.94	1.89	0.52
query25	0.47	0.29	0.08
query26	0.63	0.17	0.15
query27	0.07	0.05	0.06
query28	9.70	0.90	0.43
query29	12.57	3.98	3.35
query30	0.25	0.09	0.06
query31	2.84	0.61	0.39
query32	3.23	0.55	0.46
query33	3.06	3.11	3.13
query34	16.08	5.43	4.75
query35	4.84	4.76	4.80
query36	0.67	0.50	0.49
query37	0.09	0.06	0.07
query38	0.07	0.05	0.04
query39	0.03	0.02	0.03
query40	0.17	0.15	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 103.96 s
Total hot run time: 29.24 s

@sollhui sollhui force-pushed the parallel_begin_and_commit_txn branch from d96cd7a to 9677021 Compare June 18, 2025 16:02
@sollhui
Copy link
Contributor Author

sollhui commented Jun 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33661 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9677021c8380b9960b796d19b34ad794f2e655ad, data reload: false

------ Round 1 ----------------------------------
q1	17607	5151	4913	4913
q2	1944	297	182	182
q3	10280	1284	680	680
q4	10217	997	541	541
q5	7537	2346	2281	2281
q6	180	161	130	130
q7	883	719	610	610
q8	9308	1237	1096	1096
q9	6797	5120	5138	5120
q10	6884	2383	1976	1976
q11	480	285	283	283
q12	344	344	209	209
q13	17804	3606	3080	3080
q14	232	236	214	214
q15	562	481	480	480
q16	436	425	368	368
q17	607	835	398	398
q18	7432	7166	7030	7030
q19	1226	930	540	540
q20	326	344	243	243
q21	3844	3161	2323	2323
q22	1070	1039	964	964
Total cold run time: 106000 ms
Total hot run time: 33661 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5002	4980	4996	4980
q2	246	328	233	233
q3	2118	2646	2326	2326
q4	1328	1780	1341	1341
q5	4175	4103	4396	4103
q6	214	175	136	136
q7	1988	1886	1767	1767
q8	2543	2506	2446	2446
q9	7017	7152	7078	7078
q10	3064	3274	2815	2815
q11	583	520	494	494
q12	688	757	638	638
q13	3439	3910	3220	3220
q14	279	309	271	271
q15	517	478	481	478
q16	440	485	447	447
q17	1133	1474	1377	1377
q18	7619	7131	6981	6981
q19	766	750	827	750
q20	1920	1971	1817	1817
q21	4729	4236	4255	4236
q22	1059	1015	969	969
Total cold run time: 50867 ms
Total hot run time: 48903 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184932 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9677021c8380b9960b796d19b34ad794f2e655ad, data reload: false

query1	993	391	388	388
query2	6538	1841	1788	1788
query3	6749	222	225	222
query4	25996	23540	22863	22863
query5	4323	624	465	465
query6	306	211	190	190
query7	4627	485	277	277
query8	267	233	221	221
query9	8633	2604	2607	2604
query10	462	377	283	283
query11	15590	15011	14848	14848
query12	159	110	108	108
query13	1660	518	430	430
query14	9397	6142	6071	6071
query15	202	194	167	167
query16	7539	638	467	467
query17	1203	754	584	584
query18	2049	420	316	316
query19	212	192	164	164
query20	123	124	121	121
query21	216	137	110	110
query22	4169	4190	4100	4100
query23	33859	33124	32946	32946
query24	8455	2352	2349	2349
query25	534	456	403	403
query26	1241	274	151	151
query27	2739	512	353	353
query28	4277	2114	2104	2104
query29	734	547	429	429
query30	292	225	189	189
query31	935	842	744	744
query32	74	62	64	62
query33	549	363	302	302
query34	800	879	532	532
query35	763	793	731	731
query36	944	985	862	862
query37	107	100	78	78
query38	4091	4185	3982	3982
query39	1480	1437	1384	1384
query40	204	118	107	107
query41	65	63	58	58
query42	133	120	121	120
query43	508	490	462	462
query44	1283	825	828	825
query45	174	184	174	174
query46	844	1018	621	621
query47	1781	1806	1716	1716
query48	384	416	299	299
query49	733	497	414	414
query50	636	684	402	402
query51	4084	4137	4137	4137
query52	105	127	96	96
query53	233	252	176	176
query54	571	566	494	494
query55	85	84	84	84
query56	305	289	287	287
query57	1178	1218	1155	1155
query58	265	256	254	254
query59	2605	2647	2535	2535
query60	335	338	297	297
query61	122	123	141	123
query62	799	735	673	673
query63	223	185	188	185
query64	4300	987	697	697
query65	4218	4122	4233	4122
query66	1086	412	307	307
query67	15885	15767	15477	15477
query68	8584	890	536	536
query69	464	318	269	269
query70	1197	1091	1076	1076
query71	477	317	303	303
query72	5507	4776	4846	4776
query73	726	664	351	351
query74	8992	8891	8632	8632
query75	3982	3235	2658	2658
query76	3742	1189	752	752
query77	782	392	284	284
query78	10005	10248	9468	9468
query79	1995	799	577	577
query80	579	510	442	442
query81	462	257	232	232
query82	426	133	96	96
query83	284	250	232	232
query84	297	110	98	98
query85	784	348	307	307
query86	332	290	293	290
query87	4454	4418	4320	4320
query88	2897	2307	2283	2283
query89	389	337	291	291
query90	1927	205	207	205
query91	142	141	112	112
query92	78	66	61	61
query93	1116	938	594	594
query94	678	392	287	287
query95	381	292	278	278
query96	496	560	277	277
query97	2760	2798	2653	2653
query98	222	210	207	207
query99	1442	1396	1300	1300
Total cold run time: 273030 ms
Total hot run time: 184932 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9677021c8380b9960b796d19b34ad794f2e655ad, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.25	0.07	0.07
query4	1.61	0.10	0.11
query5	0.42	0.43	0.42
query6	1.15	0.67	0.67
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.54	0.51
query10	0.57	0.60	0.57
query11	0.15	0.11	0.12
query12	0.15	0.11	0.11
query13	0.64	0.60	0.62
query14	0.81	0.85	0.83
query15	0.88	0.87	0.88
query16	0.38	0.39	0.38
query17	1.10	1.08	1.04
query18	0.23	0.21	0.22
query19	1.99	1.88	1.90
query20	0.01	0.01	0.02
query21	15.39	0.89	0.53
query22	0.75	1.20	0.64
query23	14.94	1.37	0.59
query24	7.31	1.12	0.42
query25	0.44	0.35	0.07
query26	0.58	0.16	0.13
query27	0.08	0.05	0.05
query28	9.24	0.92	0.46
query29	12.57	3.93	3.31
query30	0.26	0.09	0.06
query31	2.83	0.59	0.40
query32	3.26	0.57	0.47
query33	3.13	3.11	3.12
query34	15.99	5.38	4.81
query35	4.80	4.84	4.81
query36	0.68	0.51	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.83 s
Total hot run time: 29.15 s

@sollhui sollhui force-pushed the parallel_begin_and_commit_txn branch from 9677021 to 8f8b708 Compare June 19, 2025 09:45
@sollhui
Copy link
Contributor Author

sollhui commented Jun 19, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34784 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f8b7083d3aa7823552cdc4f1cb8504c49556578, data reload: false

------ Round 1 ----------------------------------
q1	17605	5201	5115	5115
q2	1941	319	221	221
q3	10392	1383	733	733
q4	10248	1028	518	518
q5	7761	2463	2446	2446
q6	193	167	135	135
q7	952	756	622	622
q8	9337	1358	1269	1269
q9	6748	5207	5232	5207
q10	6942	2410	1977	1977
q11	489	303	266	266
q12	365	373	215	215
q13	17758	3712	3119	3119
q14	238	231	217	217
q15	577	486	496	486
q16	442	433	380	380
q17	624	900	382	382
q18	7959	7281	7203	7203
q19	1390	997	595	595
q20	328	347	230	230
q21	4149	3525	2490	2490
q22	1098	1034	958	958
Total cold run time: 107536 ms
Total hot run time: 34784 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5203	5252	5161	5161
q2	250	327	233	233
q3	2213	2695	2277	2277
q4	1418	1825	1328	1328
q5	4278	4418	4449	4418
q6	235	175	131	131
q7	2050	1911	1762	1762
q8	2671	2546	2582	2546
q9	7169	7221	7124	7124
q10	3090	3284	2884	2884
q11	593	522	493	493
q12	677	772	625	625
q13	3529	3945	3295	3295
q14	303	319	290	290
q15	545	483	476	476
q16	458	500	448	448
q17	1194	1581	1367	1367
q18	7804	7597	7309	7309
q19	878	941	1146	941
q20	1998	2061	1871	1871
q21	5014	4633	4239	4239
q22	1107	1030	1018	1018
Total cold run time: 52677 ms
Total hot run time: 50236 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f8b7083d3aa7823552cdc4f1cb8504c49556578, data reload: false

query1	1010	411	411	411
query2	6539	1885	1896	1885
query3	6741	224	224	224
query4	26362	23798	23383	23383
query5	4364	611	460	460
query6	306	214	217	214
query7	4634	506	292	292
query8	261	230	212	212
query9	8607	2648	2690	2648
query10	490	354	278	278
query11	15939	15271	14873	14873
query12	171	110	107	107
query13	1662	543	425	425
query14	9126	6279	6278	6278
query15	206	201	173	173
query16	7188	652	509	509
query17	1206	751	579	579
query18	1989	421	314	314
query19	214	203	169	169
query20	128	124	118	118
query21	218	133	121	121
query22	4197	4152	4205	4152
query23	34111	33301	33322	33301
query24	8447	2396	2463	2396
query25	541	493	398	398
query26	1248	270	158	158
query27	2740	527	361	361
query28	4294	2116	2110	2110
query29	797	560	444	444
query30	289	228	203	203
query31	960	867	795	795
query32	74	69	65	65
query33	552	367	319	319
query34	809	853	564	564
query35	818	847	756	756
query36	987	1002	906	906
query37	113	103	76	76
query38	4127	4099	4048	4048
query39	1532	1414	1437	1414
query40	215	120	113	113
query41	68	71	64	64
query42	135	108	111	108
query43	521	508	489	489
query44	1352	866	887	866
query45	184	182	176	176
query46	903	1075	641	641
query47	1738	1785	1740	1740
query48	397	421	323	323
query49	738	502	433	433
query50	659	695	407	407
query51	4186	4252	4175	4175
query52	113	111	104	104
query53	234	264	194	194
query54	573	594	510	510
query55	96	92	87	87
query56	328	306	315	306
query57	1188	1204	1141	1141
query58	268	264	268	264
query59	2648	2805	2650	2650
query60	319	337	306	306
query61	130	128	126	126
query62	800	739	678	678
query63	235	203	202	202
query64	4344	1019	714	714
query65	4336	4223	4207	4207
query66	1236	441	333	333
query67	15826	15682	15485	15485
query68	8037	920	551	551
query69	528	314	277	277
query70	1217	1164	1136	1136
query71	469	340	329	329
query72	5341	4810	4758	4758
query73	698	622	354	354
query74	9185	8975	8913	8913
query75	3826	3213	2736	2736
query76	3620	1223	777	777
query77	810	411	293	293
query78	10041	10290	9413	9413
query79	1974	801	610	610
query80	611	514	492	492
query81	485	255	230	230
query82	420	128	98	98
query83	252	254	228	228
query84	234	108	86	86
query85	783	370	374	370
query86	349	316	293	293
query87	4377	4470	4396	4396
query88	2923	2304	2319	2304
query89	378	313	291	291
query90	1837	210	206	206
query91	148	138	116	116
query92	73	61	62	61
query93	1048	957	602	602
query94	676	414	303	303
query95	378	301	289	289
query96	495	556	282	282
query97	2753	2745	2641	2641
query98	235	207	198	198
query99	1445	1425	1255	1255
Total cold run time: 273413 ms
Total hot run time: 187869 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f8b7083d3aa7823552cdc4f1cb8504c49556578, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.07	0.08
query4	1.59	0.10	0.10
query5	0.46	0.42	0.42
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.59	0.53	0.52
query10	0.58	0.60	0.57
query11	0.16	0.11	0.12
query12	0.15	0.11	0.12
query13	0.63	0.62	0.60
query14	0.80	0.83	0.81
query15	0.91	0.89	0.89
query16	0.39	0.40	0.40
query17	1.06	1.05	1.09
query18	0.23	0.21	0.21
query19	1.95	1.86	1.84
query20	0.01	0.01	0.02
query21	15.41	0.92	0.56
query22	0.75	1.16	0.77
query23	14.84	1.38	0.61
query24	8.06	0.70	0.75
query25	0.48	0.12	0.15
query26	0.76	0.17	0.14
query27	0.07	0.05	0.05
query28	9.02	0.90	0.46
query29	12.56	4.04	3.36
query30	0.25	0.08	0.06
query31	2.85	0.62	0.40
query32	3.24	0.56	0.48
query33	3.12	3.10	3.16
query34	16.15	5.48	4.83
query35	4.90	4.88	4.87
query36	0.65	0.49	0.49
query37	0.08	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.19	0.15	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 104.69 s
Total hot run time: 29.82 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 45.16% (14/31) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] The consumption speed of routine load job shake severely if the FDB pressure is high
4 participants
@sollhui @Thearas @doris-robot @hello-stephen