Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](group commit) Fix cloud group commit be select strategy #39986

Merged
merged 5 commits into from
Aug 28, 2024
Merged

[Fix](group commit) Fix cloud group commit be select strategy #39986

merged 5 commits into from
Aug 28, 2024

Conversation

Yukang-Lian
Copy link
Collaborator

Proposed changes

In #35558, we optimized be select for group commit. However, we forgot to apply this strategy to cloud. This PR applys it.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37503 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef842a2046e4df8887618b57f7c9627331da8307, data reload: false

------ Round 1 ----------------------------------
q1	17615	4375	4301	4301
q2	2031	186	173	173
q3	11843	962	992	962
q4	10515	705	661	661
q5	7747	2850	2786	2786
q6	220	135	136	135
q7	972	616	600	600
q8	9312	2047	2068	2047
q9	7110	6521	6534	6521
q10	6989	2224	2182	2182
q11	458	238	241	238
q12	385	218	219	218
q13	17754	3036	3015	3015
q14	291	229	240	229
q15	521	491	499	491
q16	489	384	388	384
q17	967	683	689	683
q18	7300	6800	6713	6713
q19	1400	1077	996	996
q20	674	324	327	324
q21	4192	2831	2831	2831
q22	1124	1013	1039	1013
Total cold run time: 109909 ms
Total hot run time: 37503 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4322	4233	4233
q2	375	267	263	263
q3	2893	2616	2656	2616
q4	1902	1653	1614	1614
q5	5514	5653	5643	5643
q6	224	137	130	130
q7	2236	1815	1822	1815
q8	3289	3424	3414	3414
q9	8811	8753	8785	8753
q10	3566	3359	3365	3359
q11	622	535	507	507
q12	811	653	665	653
q13	14726	3185	3212	3185
q14	321	285	288	285
q15	544	494	514	494
q16	501	432	457	432
q17	1824	1555	1528	1528
q18	8045	7714	7925	7714
q19	1694	1641	1576	1576
q20	2166	1914	1954	1914
q21	5912	5774	5589	5589
q22	1143	1031	1012	1012
Total cold run time: 71447 ms
Total hot run time: 56729 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191538 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef842a2046e4df8887618b57f7c9627331da8307, data reload: false

query1	1244	893	856	856
query2	6302	1991	1903	1903
query3	10595	3999	3960	3960
query4	59753	26715	23138	23138
query5	5515	500	484	484
query6	436	152	166	152
query7	5786	302	310	302
query8	297	205	198	198
query9	9085	2469	2444	2444
query10	487	268	261	261
query11	16479	15060	15251	15060
query12	168	101	104	101
query13	1574	401	400	400
query14	11206	7271	7257	7257
query15	245	176	182	176
query16	7643	503	440	440
query17	1157	594	576	576
query18	2070	299	287	287
query19	290	149	147	147
query20	117	110	114	110
query21	207	103	102	102
query22	4692	4401	4337	4337
query23	34405	33852	33300	33300
query24	5901	2890	2874	2874
query25	514	406	382	382
query26	693	158	160	158
query27	1769	287	283	283
query28	3778	2033	2021	2021
query29	693	405	395	395
query30	245	155	146	146
query31	949	729	731	729
query32	86	55	53	53
query33	462	280	294	280
query34	853	476	481	476
query35	842	723	708	708
query36	1056	940	936	936
query37	136	80	82	80
query38	4019	3886	3894	3886
query39	1439	1416	1403	1403
query40	202	114	113	113
query41	45	46	45	45
query42	112	97	94	94
query43	503	483	478	478
query44	1082	750	753	750
query45	199	167	168	167
query46	1083	757	757	757
query47	1929	1763	1802	1763
query48	371	311	299	299
query49	776	439	462	439
query50	809	413	425	413
query51	7265	7107	7048	7048
query52	101	90	90	90
query53	254	183	185	183
query54	565	465	465	465
query55	84	78	78	78
query56	279	268	267	267
query57	1197	1098	1078	1078
query58	232	274	222	222
query59	3112	2898	2988	2898
query60	290	269	262	262
query61	101	99	99	99
query62	743	633	636	633
query63	217	188	186	186
query64	3320	1736	1751	1736
query65	3191	3206	3157	3157
query66	620	326	330	326
query67	15267	15160	15314	15160
query68	2953	579	590	579
query69	409	279	270	270
query70	1181	1068	1143	1068
query71	339	278	273	273
query72	2595	2119	2049	2049
query73	712	328	326	326
query74	9226	8776	8950	8776
query75	3344	2697	2721	2697
query76	1429	1023	981	981
query77	517	329	312	312
query78	9702	9075	8949	8949
query79	1022	550	539	539
query80	663	548	492	492
query81	455	220	224	220
query82	289	139	131	131
query83	170	150	148	148
query84	255	75	74	74
query85	667	288	277	277
query86	293	297	301	297
query87	4300	4260	4278	4260
query88	3118	2367	2333	2333
query89	371	283	289	283
query90	1945	204	194	194
query91	119	96	96	96
query92	62	51	52	51
query93	1021	525	531	525
query94	658	301	302	301
query95	329	258	258	258
query96	581	275	267	267
query97	3188	3034	3115	3034
query98	266	206	197	197
query99	1530	1286	1274	1274
Total cold run time: 300846 ms
Total hot run time: 191538 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef842a2046e4df8887618b57f7c9627331da8307, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.50
query6	1.12	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.48
query10	0.54	0.55	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.60	0.58
query14	0.77	0.80	0.78
query15	0.89	0.83	0.81
query16	0.38	0.37	0.39
query17	0.95	1.00	1.06
query18	0.21	0.19	0.20
query19	1.86	1.84	1.76
query20	0.01	0.01	0.01
query21	15.42	0.65	0.64
query22	4.91	6.41	2.18
query23	18.28	1.41	1.33
query24	2.08	0.23	0.21
query25	0.14	0.09	0.08
query26	0.27	0.18	0.17
query27	0.07	0.08	0.08
query28	13.24	1.00	1.00
query29	12.63	3.35	3.37
query30	0.24	0.06	0.05
query31	2.88	0.41	0.39
query32	3.26	0.49	0.47
query33	2.94	3.00	3.00
query34	17.01	4.43	4.36
query35	4.42	4.42	4.42
query36	0.66	0.49	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.14	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.93 s
Total hot run time: 30.97 s

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38314 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c6c2e7ca9507b0c742ff23e8891dde027c2a7b0, data reload: false

------ Round 1 ----------------------------------
q1	17816	4513	4396	4396
q2	2635	182	174	174
q3	11672	1116	1121	1116
q4	10672	768	773	768
q5	8131	2927	2830	2830
q6	226	140	139	139
q7	987	611	610	610
q8	9332	2053	2031	2031
q9	7218	6518	6547	6518
q10	7026	2214	2217	2214
q11	466	246	251	246
q12	408	229	225	225
q13	17767	3007	3012	3007
q14	291	236	237	236
q15	542	493	503	493
q16	494	394	389	389
q17	970	708	735	708
q18	7388	6815	6938	6815
q19	1440	1043	1048	1043
q20	678	332	331	331
q21	3956	3013	3070	3013
q22	1120	1029	1012	1012
Total cold run time: 111235 ms
Total hot run time: 38314 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4441	4318	4323	4318
q2	376	272	272	272
q3	2932	2664	2725	2664
q4	1974	1695	1702	1695
q5	5390	5412	5400	5400
q6	219	131	130	130
q7	2106	1735	1778	1735
q8	3183	3350	3358	3350
q9	8405	8385	8426	8385
q10	3443	3216	3227	3216
q11	590	494	507	494
q12	810	610	614	610
q13	12154	3068	3023	3023
q14	297	276	268	268
q15	528	478	473	473
q16	481	418	429	418
q17	1787	1487	1491	1487
q18	7641	7569	7470	7470
q19	1665	1541	1538	1538
q20	2054	1790	1816	1790
q21	5516	5318	5251	5251
q22	1134	1023	1027	1023
Total cold run time: 67126 ms
Total hot run time: 55010 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186956 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c6c2e7ca9507b0c742ff23e8891dde027c2a7b0, data reload: false

query1	912	367	358	358
query2	6443	1927	1845	1845
query3	6658	217	247	217
query4	33371	23180	23045	23045
query5	4167	507	490	490
query6	259	164	160	160
query7	4580	305	289	289
query8	264	205	217	205
query9	8680	2464	2475	2464
query10	435	280	267	267
query11	17769	15163	15105	15105
query12	155	101	98	98
query13	1632	386	365	365
query14	9264	6769	7160	6769
query15	286	177	166	166
query16	7815	424	503	424
query17	1588	583	572	572
query18	1624	297	294	294
query19	230	150	150	150
query20	119	113	121	113
query21	212	108	104	104
query22	4453	4129	4054	4054
query23	33972	33377	33878	33377
query24	11748	2896	2899	2896
query25	726	390	383	383
query26	1596	160	153	153
query27	2693	285	283	283
query28	7500	2067	2045	2045
query29	948	444	406	406
query30	304	156	154	154
query31	967	781	746	746
query32	97	55	56	55
query33	756	279	281	279
query34	987	484	489	484
query35	866	742	689	689
query36	1105	946	907	907
query37	198	77	81	77
query38	3998	3978	3889	3889
query39	1440	1417	1402	1402
query40	272	115	114	114
query41	49	44	43	43
query42	109	95	101	95
query43	506	465	473	465
query44	1242	783	750	750
query45	193	166	167	166
query46	1111	723	755	723
query47	1858	1744	1772	1744
query48	387	310	296	296
query49	1126	424	420	420
query50	815	409	409	409
query51	7145	6899	7011	6899
query52	104	90	89	89
query53	255	188	177	177
query54	900	483	447	447
query55	82	76	75	75
query56	264	260	253	253
query57	1195	1098	1041	1041
query58	232	228	247	228
query59	3003	2817	2849	2817
query60	293	266	259	259
query61	101	98	102	98
query62	811	631	641	631
query63	218	183	183	183
query64	5090	704	657	657
query65	3190	3134	3098	3098
query66	1112	341	346	341
query67	15676	15312	15031	15031
query68	3351	578	590	578
query69	379	269	280	269
query70	1115	1037	1123	1037
query71	335	275	278	275
query72	6201	4009	4113	4009
query73	746	331	334	331
query74	8963	8759	8884	8759
query75	3362	2694	2695	2694
query76	1985	982	996	982
query77	470	319	302	302
query78	9663	9067	9035	9035
query79	1022	539	554	539
query80	692	504	492	492
query81	492	229	231	229
query82	237	145	143	143
query83	171	154	155	154
query84	224	75	76	75
query85	848	283	272	272
query86	301	297	299	297
query87	4310	4322	4287	4287
query88	3553	2460	2357	2357
query89	376	289	285	285
query90	1899	198	195	195
query91	123	108	97	97
query92	60	51	52	51
query93	1026	548	544	544
query94	689	285	293	285
query95	353	266	262	262
query96	586	272	271	271
query97	3217	3075	3023	3023
query98	222	208	205	205
query99	1453	1294	1295	1294
Total cold run time: 286580 ms
Total hot run time: 186956 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c6c2e7ca9507b0c742ff23e8891dde027c2a7b0, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.50
query6	1.13	0.74	0.74
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.48	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.76	0.79	0.76
query15	0.88	0.83	0.81
query16	0.37	0.38	0.39
query17	0.95	0.95	0.97
query18	0.21	0.20	0.20
query19	1.92	1.74	1.85
query20	0.01	0.02	0.01
query21	15.40	0.67	0.66
query22	4.17	7.74	1.83
query23	18.24	1.35	1.21
query24	2.07	0.21	0.21
query25	0.15	0.08	0.07
query26	0.26	0.18	0.17
query27	0.08	0.08	0.08
query28	13.33	1.01	0.98
query29	12.61	3.38	3.34
query30	0.23	0.06	0.06
query31	2.88	0.40	0.39
query32	3.26	0.47	0.48
query33	2.97	3.00	2.99
query34	16.99	4.35	4.33
query35	4.44	4.45	4.48
query36	0.64	0.46	0.49
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.17	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.3 s
Total hot run time: 30.47 s

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38341 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3271b2f969c36775d677c265ed9908db05f14ca, data reload: false

------ Round 1 ----------------------------------
q1	17612	4446	4336	4336
q2	2023	184	180	180
q3	11609	970	1123	970
q4	10510	750	745	745
q5	7747	2936	2889	2889
q6	228	140	138	138
q7	950	609	592	592
q8	9360	2113	2109	2109
q9	7296	6590	6644	6590
q10	7020	2217	2154	2154
q11	461	247	239	239
q12	408	217	225	217
q13	17755	3025	3040	3025
q14	286	246	234	234
q15	520	483	495	483
q16	603	515	504	504
q17	1038	684	616	616
q18	7399	6979	6950	6950
q19	1398	1011	994	994
q20	680	319	332	319
q21	3888	3043	3101	3043
q22	1130	1014	1047	1014
Total cold run time: 109921 ms
Total hot run time: 38341 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4429	4332	4278	4278
q2	384	285	274	274
q3	2928	2643	2651	2643
q4	1962	1650	1657	1650
q5	5669	5693	5763	5693
q6	234	140	139	139
q7	2233	1863	1877	1863
q8	3373	3439	3472	3439
q9	8890	8945	8918	8918
q10	3616	3434	3395	3395
q11	598	540	523	523
q12	850	688	670	670
q13	13840	3177	3245	3177
q14	334	306	293	293
q15	534	482	484	482
q16	619	560	555	555
q17	1889	1554	1515	1515
q18	8301	7863	7969	7863
q19	1764	1649	1578	1578
q20	2184	1941	1906	1906
q21	5851	5461	5615	5461
q22	1136	1065	1048	1048
Total cold run time: 71618 ms
Total hot run time: 57363 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192907 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3271b2f969c36775d677c265ed9908db05f14ca, data reload: false

query1	1265	892	860	860
query2	6289	2026	1962	1962
query3	10754	4128	4096	4096
query4	59938	23807	23415	23415
query5	5518	485	498	485
query6	422	171	195	171
query7	5868	308	298	298
query8	297	210	205	205
query9	8842	2554	2531	2531
query10	492	288	262	262
query11	17733	14954	15353	14954
query12	151	104	100	100
query13	1580	400	419	400
query14	11513	6600	7122	6600
query15	256	179	194	179
query16	7592	517	505	505
query17	1186	569	547	547
query18	2175	296	305	296
query19	287	156	158	156
query20	122	116	114	114
query21	208	102	103	102
query22	4552	4335	4551	4335
query23	34352	33754	33395	33395
query24	5869	2891	2958	2891
query25	534	374	378	374
query26	690	160	157	157
query27	1765	295	295	295
query28	3964	2154	2147	2147
query29	711	423	411	411
query30	243	157	156	156
query31	920	723	750	723
query32	80	54	55	54
query33	463	292	277	277
query34	878	498	485	485
query35	871	717	732	717
query36	1069	930	941	930
query37	158	98	98	98
query38	4039	3890	3884	3884
query39	1466	1382	1421	1382
query40	205	120	116	116
query41	48	45	45	45
query42	118	101	97	97
query43	528	490	489	489
query44	1121	750	762	750
query45	200	163	162	162
query46	1093	785	773	773
query47	1872	1806	1834	1806
query48	378	315	316	315
query49	773	442	463	442
query50	863	421	428	421
query51	7189	7061	7055	7055
query52	105	95	92	92
query53	256	190	188	188
query54	585	458	462	458
query55	82	83	81	81
query56	285	268	253	253
query57	1214	1059	1090	1059
query58	235	237	246	237
query59	3018	2835	2883	2835
query60	308	281	292	281
query61	121	124	120	120
query62	746	653	646	646
query63	225	190	197	190
query64	2932	698	657	657
query65	3230	3147	3157	3147
query66	685	340	343	340
query67	15494	15260	15141	15141
query68	2961	596	599	596
query69	399	296	281	281
query70	1189	1140	1100	1100
query71	360	274	279	274
query72	5979	3987	4051	3987
query73	792	336	334	334
query74	9162	8783	8903	8783
query75	3360	2701	2730	2701
query76	1405	1057	1007	1007
query77	599	348	338	338
query78	9748	9076	9160	9076
query79	1064	565	531	531
query80	711	533	510	510
query81	469	242	234	234
query82	249	145	142	142
query83	172	149	149	149
query84	262	77	79	77
query85	685	293	339	293
query86	300	298	281	281
query87	4352	4210	4213	4210
query88	3344	2311	2318	2311
query89	384	285	290	285
query90	1922	199	195	195
query91	127	98	101	98
query92	63	51	57	51
query93	1087	537	541	537
query94	720	292	310	292
query95	318	269	269	269
query96	598	269	269	269
query97	3213	3095	3088	3088
query98	222	207	210	207
query99	1501	1284	1256	1256
Total cold run time: 306738 ms
Total hot run time: 192907 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3271b2f969c36775d677c265ed9908db05f14ca, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.09	0.09
query5	0.51	0.49	0.48
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.48
query10	0.54	0.53	0.56
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.09	2.04	2.13
query15	0.84	0.81	0.82
query16	0.38	0.36	0.37
query17	1.02	0.96	0.99
query18	0.21	0.20	0.21
query19	1.90	1.79	1.87
query20	0.02	0.01	0.01
query21	15.39	0.66	0.66
query22	4.24	6.22	2.39
query23	18.28	1.48	1.21
query24	1.88	0.30	0.22
query25	0.14	0.07	0.08
query26	0.28	0.18	0.19
query27	0.08	0.07	0.07
query28	13.21	1.02	1.00
query29	12.63	3.27	3.31
query30	0.24	0.06	0.06
query31	2.89	0.41	0.39
query32	3.22	0.49	0.47
query33	3.03	3.00	3.04
query34	17.00	4.46	4.41
query35	4.52	4.46	4.51
query36	0.75	0.46	0.47
query37	0.20	0.17	0.17
query38	0.16	0.15	0.16
query39	0.04	0.04	0.04
query40	0.15	0.14	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.72 s
Total hot run time: 32.41 s

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37968 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc78120113ceb55d17390474a526712d9e976420, data reload: false

------ Round 1 ----------------------------------
q1	17860	5103	4279	4279
q2	2034	197	173	173
q3	11673	950	1143	950
q4	10384	862	739	739
q5	7765	2855	2687	2687
q6	232	139	138	138
q7	978	620	608	608
q8	9341	2120	2071	2071
q9	7228	6547	6536	6536
q10	6992	2227	2165	2165
q11	454	239	244	239
q12	393	225	226	225
q13	18866	3031	3035	3031
q14	270	239	240	239
q15	524	477	485	477
q16	578	519	495	495
q17	1000	666	704	666
q18	7491	6802	6778	6778
q19	1382	1073	1083	1073
q20	697	351	348	348
q21	3894	3031	3140	3031
q22	1123	1054	1020	1020
Total cold run time: 111159 ms
Total hot run time: 37968 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4527	4285	4273	4273
q2	381	281	276	276
q3	2829	2649	2682	2649
q4	1953	1674	1690	1674
q5	5672	5684	5811	5684
q6	236	138	147	138
q7	2177	1851	1841	1841
q8	3290	3409	3459	3409
q9	8863	8831	8813	8813
q10	3622	3406	3409	3406
q11	614	509	527	509
q12	850	664	637	637
q13	12992	3162	3257	3162
q14	328	310	292	292
q15	539	483	476	476
q16	630	549	561	549
q17	1851	1575	1558	1558
q18	8091	7854	7851	7851
q19	1706	1385	1603	1385
q20	2133	1896	1925	1896
q21	5807	5503	5529	5503
q22	1227	1062	1048	1048
Total cold run time: 70318 ms
Total hot run time: 57029 ms

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 192464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc78120113ceb55d17390474a526712d9e976420, data reload: false

query1	1262	884	856	856
query2	6292	1980	1953	1953
query3	10739	4060	4041	4041
query4	60108	25110	23247	23247
query5	5353	502	501	501
query6	403	163	176	163
query7	5779	295	296	295
query8	290	206	204	204
query9	8939	2498	2486	2486
query10	494	270	251	251
query11	17996	15081	15326	15081
query12	165	105	100	100
query13	1559	397	381	381
query14	11330	7482	6886	6886
query15	214	188	171	171
query16	7543	505	519	505
query17	1138	590	562	562
query18	1985	285	291	285
query19	292	150	144	144
query20	126	110	117	110
query21	206	111	107	107
query22	4824	4384	4850	4384
query23	34136	34180	33145	33145
query24	5960	2890	2840	2840
query25	488	371	370	370
query26	682	154	154	154
query27	1759	291	282	282
query28	3825	2145	2140	2140
query29	676	433	449	433
query30	236	157	157	157
query31	963	745	791	745
query32	83	57	54	54
query33	465	289	279	279
query34	874	477	478	477
query35	866	742	726	726
query36	1078	920	931	920
query37	149	89	94	89
query38	3914	3820	3797	3797
query39	1459	1410	1379	1379
query40	196	112	114	112
query41	43	41	45	41
query42	116	96	98	96
query43	530	472	483	472
query44	1124	749	743	743
query45	201	168	165	165
query46	1087	738	744	738
query47	1920	1836	1834	1834
query48	368	290	297	290
query49	751	422	436	422
query50	829	403	414	403
query51	7174	7135	7058	7058
query52	96	89	87	87
query53	249	178	178	178
query54	560	451	442	442
query55	78	77	78	77
query56	277	270	312	270
query57	1192	1049	1087	1049
query58	215	243	230	230
query59	3098	2881	2773	2773
query60	292	261	268	261
query61	101	99	101	99
query62	753	668	657	657
query63	217	185	186	185
query64	2757	665	703	665
query65	3230	3125	3131	3125
query66	680	339	331	331
query67	15617	15407	15206	15206
query68	3004	577	572	572
query69	403	273	275	273
query70	1168	1116	1047	1047
query71	359	273	268	268
query72	4948	4140	4000	4000
query73	757	350	334	334
query74	9065	8798	8832	8798
query75	3396	2652	2689	2652
query76	1440	1078	994	994
query77	541	318	319	318
query78	11394	9357	9126	9126
query79	1031	556	540	540
query80	696	563	524	524
query81	464	238	238	238
query82	240	156	149	149
query83	172	173	158	158
query84	260	78	75	75
query85	679	281	277	277
query86	309	306	309	306
query87	4439	4261	4220	4220
query88	3119	2355	2455	2355
query89	366	285	287	285
query90	1944	199	192	192
query91	119	99	100	99
query92	61	54	54	54
query93	1044	551	538	538
query94	732	300	303	300
query95	339	258	269	258
query96	594	274	270	270
query97	3171	3066	3052	3052
query98	221	202	258	202
query99	1529	1289	1281	1281
Total cold run time: 306405 ms
Total hot run time: 192464 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc78120113ceb55d17390474a526712d9e976420, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.50	0.49
query6	1.14	0.75	0.74
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.55	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.10	2.15	2.07
query15	0.84	0.81	0.81
query16	0.38	0.38	0.37
query17	0.95	1.00	1.02
query18	0.21	0.20	0.20
query19	1.84	1.85	1.86
query20	0.02	0.01	0.01
query21	15.40	0.69	0.66
query22	4.08	7.48	1.74
query23	18.28	1.42	1.27
query24	2.06	0.23	0.22
query25	0.17	0.09	0.08
query26	0.26	0.19	0.17
query27	0.08	0.07	0.07
query28	13.30	1.02	1.01
query29	12.90	3.33	3.27
query30	0.24	0.06	0.05
query31	2.88	0.43	0.40
query32	3.23	0.49	0.47
query33	2.95	3.01	3.02
query34	16.83	4.43	4.42
query35	4.49	4.48	4.50
query36	0.65	0.50	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.63 s
Total hot run time: 32.02 s

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38200 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 613d2b5eb8a3f79de5702b7efdf6945e7d377c4f, data reload: false

------ Round 1 ----------------------------------
q1	14310	4439	4316	4316
q2	1319	181	178	178
q3	10101	1108	1035	1035
q4	8135	693	730	693
q5	4761	2866	2843	2843
q6	228	141	142	141
q7	955	624	611	611
q8	4240	2035	2085	2035
q9	6648	6554	6536	6536
q10	2974	2193	2202	2193
q11	402	261	254	254
q12	400	225	223	223
q13	17179	2988	3026	2988
q14	278	249	251	249
q15	539	497	502	497
q16	583	515	493	493
q17	972	646	708	646
q18	7545	6890	6979	6890
q19	1202	1058	1012	1012
q20	604	345	336	336
q21	3806	3123	3010	3010
q22	1119	1021	1060	1021
Total cold run time: 88300 ms
Total hot run time: 38200 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4343	4271	4271
q2	384	291	282	282
q3	2944	2693	2695	2693
q4	1960	1658	1598	1598
q5	5382	5386	5416	5386
q6	218	131	131	131
q7	2116	1734	1776	1734
q8	3190	3381	3332	3332
q9	8453	8376	8449	8376
q10	3433	3235	3173	3173
q11	605	531	519	519
q12	784	620	624	620
q13	13537	3087	3025	3025
q14	308	278	272	272
q15	519	473	481	473
q16	609	562	584	562
q17	1801	1521	1496	1496
q18	7900	7533	7623	7533
q19	1671	1541	1582	1541
q20	2080	1842	1809	1809
q21	5541	5263	5105	5105
q22	1102	1043	1048	1043
Total cold run time: 68911 ms
Total hot run time: 54974 ms

dataroaring
dataroaring approved these changes Aug 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
@doris-robot
Copy link

TPC-DS: Total hot run time: 188093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 613d2b5eb8a3f79de5702b7efdf6945e7d377c4f, data reload: false

query1	917	379	370	370
query2	6453	1964	1959	1959
query3	6643	210	219	210
query4	33680	23111	23138	23111
query5	4136	515	489	489
query6	254	164	161	161
query7	4575	301	293	293
query8	259	212	224	212
query9	8679	2494	2484	2484
query10	427	277	271	271
query11	17832	15046	15065	15046
query12	152	100	100	100
query13	1617	373	383	373
query14	9436	6850	7287	6850
query15	281	190	178	178
query16	8120	468	480	468
query17	1592	585	567	567
query18	1926	305	295	295
query19	307	151	150	150
query20	119	111	110	110
query21	216	107	107	107
query22	4510	4146	4084	4084
query23	33976	33562	33234	33234
query24	10996	2872	2866	2866
query25	662	408	413	408
query26	1517	168	163	163
query27	2805	291	284	284
query28	7761	2139	2123	2123
query29	934	433	423	423
query30	296	160	154	154
query31	978	785	769	769
query32	105	59	62	59
query33	786	291	300	291
query34	965	480	488	480
query35	859	727	726	726
query36	1087	916	946	916
query37	177	95	93	93
query38	3978	3937	3927	3927
query39	1450	1418	1386	1386
query40	270	124	120	120
query41	52	48	49	48
query42	118	100	101	100
query43	506	471	474	471
query44	1190	764	761	761
query45	195	170	175	170
query46	1105	731	762	731
query47	1927	1801	1771	1771
query48	374	302	295	295
query49	1191	445	433	433
query50	815	415	419	415
query51	7295	7078	7048	7048
query52	103	89	90	89
query53	263	190	185	185
query54	1013	475	476	475
query55	82	78	78	78
query56	281	271	272	271
query57	1218	1069	1087	1069
query58	249	227	235	227
query59	3084	2981	2897	2897
query60	310	287	275	275
query61	124	121	124	121
query62	854	673	667	667
query63	231	186	188	186
query64	5278	837	655	655
query65	3444	3123	3177	3123
query66	1396	337	357	337
query67	15442	15394	15271	15271
query68	4647	573	560	560
query69	432	278	279	278
query70	1129	1121	1058	1058
query71	417	275	274	274
query72	6899	3985	3946	3946
query73	760	334	335	334
query74	9121	8892	8791	8791
query75	3464	2666	2692	2666
query76	2805	1060	948	948
query77	512	323	319	319
query78	10277	9131	9724	9131
query79	1949	536	534	534
query80	1041	500	501	500
query81	580	243	232	232
query82	688	156	145	145
query83	222	151	156	151
query84	226	82	74	74
query85	1362	302	281	281
query86	440	290	293	290
query87	4373	4291	4296	4291
query88	3357	2329	2336	2329
query89	381	286	295	286
query90	1812	192	193	192
query91	124	139	107	107
query92	73	53	51	51
query93	1115	550	542	542
query94	888	301	290	290
query95	360	261	266	261
query96	597	276	267	267
query97	3207	3044	3104	3044
query98	220	208	203	203
query99	1671	1284	1239	1239
Total cold run time: 295022 ms
Total hot run time: 188093 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 613d2b5eb8a3f79de5702b7efdf6945e7d377c4f, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.70	0.09	0.08
query5	0.52	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.50	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.64	0.59	0.59
query14	2.05	2.04	2.11
query15	0.82	0.82	0.82
query16	0.38	0.38	0.35
query17	1.06	1.00	0.98
query18	0.21	0.20	0.20
query19	1.80	1.74	1.70
query20	0.02	0.01	0.01
query21	15.39	0.67	0.67
query22	3.89	7.19	1.87
query23	18.29	1.41	1.30
query24	2.15	0.24	0.22
query25	0.15	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.21	1.00	1.00
query29	12.64	3.28	3.24
query30	0.25	0.06	0.06
query31	2.89	0.41	0.38
query32	3.24	0.47	0.46
query33	2.97	2.96	3.03
query34	16.95	4.40	4.43
query35	4.46	4.40	4.38
query36	0.66	0.47	0.48
query37	0.19	0.15	0.16
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.4 s
Total hot run time: 31.79 s

@dataroaring dataroaring merged commit 3770e42 into apache:master Aug 28, 2024
27 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Aug 28, 2024
@Yukang-Lian @dataroaring
[Fix](group commit) Fix cloud group commit be select strategy (#39986)
8d0cd03 
## Proposed changes

In #35558, we optimized be select for group commit. However, we forgot
to apply this strategy to cloud. This PR applys it.

<!--Describe your changes.-->
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Sep 3, 2024
@Yukang-Lian
[Fix](group commit) Fix cloud group commit be select strategy (apache…
b963b09 
…#39986)

In apache#35558, we optimized be select for group commit. However, we forgot
to apply this strategy to cloud. This PR applys it.

<!--Describe your changes.-->
@Yukang-Lian Yukang-Lian mentioned this pull request Sep 3, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 5, 2024
@Yukang-Lian
[cherry-pick](branch-2.1) Fix some group commit fault (#40319)
52393f8 
## Proposed changes

Pick #39986 #40120

<!--Describe your changes.-->
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@mymeiyi mymeiyi mymeiyi approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Yukang-Lian @doris-robot @mymeiyi @dataroaring @yiguolei