Skip to content

Conversation

@dataroaring
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39572 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7571512bc1995f7156c3ce5683ef103ff61954a3, data reload: false

------ Round 1 ----------------------------------
q1	17570	5275	5539	5275
q2	2041	144	131	131
q3	10604	984	979	979
q4	4957	794	833	794
q5	8028	3033	3079	3033
q6	195	127	127	127
q7	1129	750	759	750
q8	9583	2110	2094	2094
q9	7849	6626	6433	6433
q10	8104	2456	2445	2445
q11	414	227	226	226
q12	766	299	292	292
q13	17979	3401	3349	3349
q14	289	260	266	260
q15	554	518	508	508
q16	483	404	417	404
q17	947	545	500	500
q18	7589	6943	6719	6719
q19	1563	1450	1482	1450
q20	618	346	328	328
q21	6966	3243	3172	3172
q22	823	303	308	303
Total cold run time: 109051 ms
Total hot run time: 39572 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5667	5305	5363	5305
q2	331	211	208	208
q3	3338	3297	3292	3292
q4	2105	2090	2102	2090
q5	6108	6026	6033	6026
q6	202	121	117	117
q7	2240	1917	1912	1912
q8	3251	3408	3424	3408
q9	8959	8829	8898	8829
q10	5391	3879	3881	3879
q11	566	445	437	437
q12	792	664	618	618
q13	16913	3189	3199	3189
q14	279	258	264	258
q15	547	501	507	501
q16	522	475	465	465
q17	1874	1815	1905	1815
q18	9657	19403	9614	9614
q19	23868	1593	1556	1556
q20	4619	1941	1921	1921
q21	14638	5425	5486	5425
q22	994	548	513	513
Total cold run time: 112861 ms
Total hot run time: 61378 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175596 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7571512bc1995f7156c3ce5683ef103ff61954a3, data reload: false

query1	942	325	328	325
query2	6565	2048	1823	1823
query3	6697	200	197	197
query4	32289	22183	22192	22183
query5	4451	436	445	436
query6	257	151	146	146
query7	4602	270	261	261
query8	220	175	182	175
query9	8332	2513	2508	2508
query10	419	229	232	229
query11	17026	15541	15559	15541
query12	120	68	65	65
query13	1676	387	371	371
query14	9731	6781	6668	6668
query15	215	181	185	181
query16	5693	251	248	248
query17	959	471	468	468
query18	1796	252	253	252
query19	179	129	131	129
query20	66	68	60	60
query21	190	133	132	132
query22	4856	4679	4780	4679
query23	31663	30882	30799	30799
query24	12718	2811	2857	2811
query25	587	312	310	310
query26	1950	144	139	139
query27	3172	282	290	282
query28	7371	1816	1802	1802
query29	2268	606	589	589
query30	284	135	136	135
query31	936	751	744	744
query32	85	52	46	46
query33	714	208	202	202
query34	1175	444	467	444
query35	914	758	787	758
query36	1319	1093	1150	1093
query37	180	59	54	54
query38	3350	3214	3226	3214
query39	1305	1249	1251	1249
query40	344	83	81	81
query41	36	34	34	34
query42	89	80	87	80
query43	506	475	474	474
query44	1076	697	685	685
query45	191	174	177	174
query46	1066	653	639	639
query47	1635	1472	1552	1472
query48	396	303	312	303
query49	1225	275	278	275
query50	682	300	309	300
query51	5290	5243	5198	5198
query52	83	77	69	69
query53	326	254	264	254
query54	242	181	176	176
query55	83	79	77	77
query56	174	163	161	161
query57	971	917	876	876
query58	192	153	158	153
query59	2799	2516	2629	2516
query60	207	183	171	171
query61	83	82	80	80
query62	620	378	368	368
query63	278	266	253	253
query64	6004	1777	1736	1736
query65	3335	3247	3244	3244
query66	1511	324	310	310
query67	15443	15123	15420	15123
query68	12890	521	532	521
query69	606	285	299	285
query70	1686	1491	1454	1454
query71	545	215	213	213
query72	4900	2836	2818	2818
query73	2868	320	305	305
query74	7117	6358	6317	6317
query75	5091	2300	2259	2259
query76	6704	1052	1070	1052
query77	672	225	229	225
query78	9211	8596	8678	8596
query79	2264	509	500	500
query80	546	316	307	307
query81	453	200	201	200
query82	205	81	85	81
query83	133	109	118	109
query84	270	68	68	68
query85	1068	344	333	333
query86	373	383	372	372
query87	3449	3340	3276	3276
query88	3018	2213	2212	2212
query89	428	371	338	338
query90	2010	188	191	188
query91	147	112	115	112
query92	51	44	42	42
query93	1526	423	445	423
query94	1230	160	156	156
query95	493	457	439	439
query96	610	332	321	321
query97	4252	4145	4173	4145
query98	201	203	188	188
query99	985	711	713	711
Total cold run time: 298161 ms
Total hot run time: 175596 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7571512bc1995f7156c3ce5683ef103ff61954a3, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.02	0.02
query3	0.22	0.05	0.05
query4	1.67	0.10	0.08
query5	0.53	0.53	0.53
query6	1.29	0.62	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.52	0.48
query10	0.55	0.55	0.56
query11	0.12	0.08	0.08
query12	0.11	0.09	0.08
query13	0.61	0.61	0.61
query14	0.78	0.80	0.82
query15	0.79	0.78	0.77
query16	0.40	0.37	0.37
query17	1.02	0.99	1.01
query18	0.24	0.26	0.24
query19	1.89	1.79	1.77
query20	0.01	0.00	0.01
query21	15.41	0.59	0.57
query22	2.31	2.42	2.17
query23	17.09	0.89	0.75
query24	2.56	1.06	0.21
query25	0.34	0.31	0.16
query26	0.37	0.14	0.14
query27	0.05	0.05	0.06
query28	11.99	0.77	0.76
query29	12.48	3.21	3.32
query30	0.59	0.49	0.49
query31	2.78	0.36	0.35
query32	3.35	0.48	0.48
query33	3.23	3.16	3.20
query34	15.80	4.30	4.33
query35	4.34	4.37	4.31
query36	1.10	1.07	1.08
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.08	0.01	0.01
query42	0.03	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.11 s
Total hot run time: 30.53 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7571512bc1995f7156c3ce5683ef103ff61954a3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@dataroaring dataroaring force-pushed the ccr_largebinlog_master branch from 7571512 to 36d5e6e Compare January 26, 2024 11:09
@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring
Copy link
Contributor Author

run buildall

@dataroaring
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37717 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 645ac6538e107d2f3a1d60484b17db41788c1996, data reload: false

------ Round 1 ----------------------------------
q1	17680	5607	5092	5092
q2	2062	137	132	132
q3	10716	925	957	925
q4	4716	682	744	682
q5	7694	2793	2828	2793
q6	184	119	121	119
q7	1141	728	713	713
q8	9358	2030	2043	2030
q9	7245	6385	6367	6367
q10	8100	2408	2431	2408
q11	407	208	209	208
q12	777	282	284	282
q13	18004	3287	3316	3287
q14	268	243	247	243
q15	532	521	489	489
q16	471	407	407	407
q17	944	525	513	513
q18	7153	6103	5988	5988
q19	1609	1407	1449	1407
q20	628	336	338	336
q21	7298	3215	3002	3002
q22	816	306	294	294
Total cold run time: 107803 ms
Total hot run time: 37717 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5417	5073	4981	4981
q2	341	246	234	234
q3	2984	2658	2681	2658
q4	1838	1631	1540	1540
q5	5352	5372	5166	5166
q6	200	114	117	114
q7	2141	1830	1747	1747
q8	3138	3226	3188	3188
q9	8649	8695	11590	8695
q10	5401	3400	3455	3400
q11	546	483	462	462
q12	758	629	607	607
q13	16999	3137	3090	3090
q14	286	264	257	257
q15	536	489	494	489
q16	514	469	483	469
q17	2126	1655	1770	1655
q18	8906	11380	8794	8794
q19	24057	1587	1513	1513
q20	4618	1901	1913	1901
q21	15736	4991	4869	4869
q22	1146	486	475	475
Total cold run time: 111689 ms
Total hot run time: 56304 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174078 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 645ac6538e107d2f3a1d60484b17db41788c1996, data reload: false

query1	919	331	324	324
query2	6597	2037	1810	1810
query3	6703	196	193	193
query4	33625	22222	22013	22013
query5	4460	353	362	353
query6	260	156	160	156
query7	4626	263	256	256
query8	239	176	177	176
query9	8608	2289	2283	2283
query10	410	198	197	197
query11	17902	15434	15421	15421
query12	117	68	64	64
query13	1672	389	371	371
query14	9147	6475	6484	6475
query15	211	173	181	173
query16	5695	274	248	248
query17	946	505	473	473
query18	1821	257	245	245
query19	172	130	129	129
query20	73	73	70	70
query21	198	135	131	131
query22	4837	4666	4610	4610
query23	31189	30418	30411	30411
query24	11501	2790	2773	2773
query25	549	312	293	293
query26	888	137	139	137
query27	3038	277	285	277
query28	6399	1859	1835	1835
query29	917	612	590	590
query30	279	132	139	132
query31	907	719	703	703
query32	78	53	49	49
query33	659	199	209	199
query34	1094	446	453	446
query35	841	766	741	741
query36	1394	1307	1222	1222
query37	93	59	60	59
query38	3266	3133	3150	3133
query39	1331	1256	1233	1233
query40	196	87	79	79
query41	37	34	34	34
query42	82	76	79	76
query43	519	471	483	471
query44	1069	683	678	678
query45	188	177	173	173
query46	1054	658	640	640
query47	1594	1516	1506	1506
query48	383	309	303	303
query49	1114	305	279	279
query50	688	312	310	310
query51	5330	5194	5187	5187
query52	81	85	76	76
query53	320	268	257	257
query54	248	183	184	183
query55	82	74	77	74
query56	176	156	176	156
query57	932	900	917	900
query58	182	160	156	156
query59	2777	2459	2560	2459
query60	197	174	174	174
query61	81	88	81	81
query62	631	360	365	360
query63	276	266	270	266
query64	4311	1812	1762	1762
query65	3301	3218	3232	3218
query66	1177	321	313	313
query67	15414	14979	15005	14979
query68	12682	518	501	501
query69	615	286	305	286
query70	1889	1487	1564	1487
query71	482	208	216	208
query72	4960	2819	2831	2819
query73	3044	315	308	308
query74	6804	6337	6283	6283
query75	4916	2289	2299	2289
query76	6324	1057	955	955
query77	675	235	229	229
query78	9070	8834	8751	8751
query79	1017	483	492	483
query80	548	323	316	316
query81	469	203	200	200
query82	198	81	85	81
query83	138	112	110	110
query84	277	74	72	72
query85	1097	335	334	334
query86	402	416	412	412
query87	3430	3240	3261	3240
query88	2934	2150	2166	2150
query89	435	348	354	348
query90	1960	188	180	180
query91	146	118	136	118
query92	48	42	41	41
query93	1479	445	433	433
query94	1243	165	159	159
query95	507	449	450	449
query96	608	313	309	309
query97	4295	4142	4122	4122
query98	200	192	189	189
query99	963	686	662	662
Total cold run time: 289956 ms
Total hot run time: 174078 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 645ac6538e107d2f3a1d60484b17db41788c1996, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.52	0.52
query6	1.18	0.62	0.63
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.47
query10	0.55	0.54	0.55
query11	0.12	0.08	0.09
query12	0.11	0.08	0.09
query13	0.60	0.60	0.61
query14	0.78	0.79	0.80
query15	0.79	0.79	0.77
query16	0.39	0.38	0.39
query17	0.98	0.99	1.03
query18	0.20	0.27	0.23
query19	1.88	1.81	1.78
query20	0.01	0.00	0.01
query21	15.40	0.58	0.59
query22	2.09	2.74	2.10
query23	17.28	0.77	0.74
query24	2.61	1.39	0.73
query25	0.35	0.22	0.24
query26	0.50	0.14	0.13
query27	0.04	0.05	0.05
query28	10.95	0.84	0.83
query29	12.50	3.29	3.30
query30	0.61	0.54	0.49
query31	2.79	0.34	0.36
query32	3.36	0.48	0.48
query33	3.21	3.20	3.22
query34	15.77	4.39	4.25
query35	4.38	4.29	4.29
query36	1.12	1.08	1.06
query37	0.06	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.15
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.06 s
Total hot run time: 31.14 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 645ac6538e107d2f3a1d60484b17db41788c1996 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 28, 2024
@dataroaring dataroaring merged commit 461e649 into apache:master Jan 28, 2024
yiguolei pushed a commit that referenced this pull request Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants