Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Nereids) add logical project to prevent extra wrong column #30459

Merged
merged 1 commit into from Feb 1, 2024

Conversation

iwanttobepowerful
Copy link
Contributor

Proposed changes

Issue Number: close #30264

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37530 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false

------ Round 1 ----------------------------------
q1	17708	4792	5081	4792
q2	2046	144	135	135
q3	10705	925	938	925
q4	4746	753	738	738
q5	7678	2901	2856	2856
q6	184	119	119	119
q7	1095	746	706	706
q8	9370	2028	1981	1981
q9	7222	6313	6333	6313
q10	8069	2459	2401	2401
q11	420	196	196	196
q12	767	278	273	273
q13	18027	3286	3291	3286
q14	274	239	246	239
q15	524	486	492	486
q16	464	396	392	392
q17	924	508	568	508
q18	6870	6153	6175	6153
q19	1568	1367	1378	1367
q20	621	348	333	333
q21	6796	3061	3025	3025
q22	815	308	306	306
Total cold run time: 106893 ms
Total hot run time: 37530 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5121	5031	5063	5031
q2	326	229	236	229
q3	2985	2630	2617	2617
q4	1817	1482	1502	1482
q5	5314	5182	5341	5182
q6	196	117	113	113
q7	2153	1764	1823	1764
q8	3118	3152	3260	3152
q9	8684	8623	9555	8623
q10	5656	3451	3475	3451
q11	563	466	484	466
q12	750	583	597	583
q13	16949	3096	3117	3096
q14	273	262	272	262
q15	528	493	487	487
q16	505	472	464	464
q17	1867	1775	1708	1708
q18	8719	16202	8620	8620
q19	26820	1544	1538	1538
q20	5198	1899	1893	1893
q21	14349	4918	4834	4834
q22	1021	487	448	448
Total cold run time: 112912 ms
Total hot run time: 56043 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172995 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false

query1	938	335	328	328
query2	6568	2057	1867	1867
query3	6709	197	195	195
query4	32414	22187	22078	22078
query5	4457	357	351	351
query6	254	166	156	156
query7	4610	270	264	264
query8	241	182	183	182
query9	8634	2293	2286	2286
query10	407	212	199	199
query11	16441	15335	15284	15284
query12	122	71	66	66
query13	1705	385	388	385
query14	9196	6333	6493	6333
query15	211	183	183	183
query16	5714	254	250	250
query17	951	487	467	467
query18	1800	259	251	251
query19	190	133	131	131
query20	70	74	65	65
query21	197	126	135	126
query22	4910	4720	4573	4573
query23	31222	30470	30419	30419
query24	12476	2798	2790	2790
query25	599	301	313	301
query26	1951	144	140	140
query27	3177	287	283	283
query28	7658	1817	1821	1817
query29	2259	622	617	617
query30	284	134	147	134
query31	900	719	718	718
query32	78	52	51	51
query33	692	215	203	203
query34	1192	458	471	458
query35	859	763	760	760
query36	1329	1256	1217	1217
query37	95	59	58	58
query38	3244	3190	3149	3149
query39	1317	1259	1259	1259
query40	341	92	85	85
query41	41	35	36	35
query42	93	82	80	80
query43	530	470	493	470
query44	1126	682	687	682
query45	196	181	175	175
query46	1068	668	659	659
query47	1609	1527	1526	1526
query48	385	303	311	303
query49	1218	287	286	286
query50	677	313	298	298
query51	5384	5145	5198	5145
query52	89	88	77	77
query53	323	259	273	259
query54	246	193	186	186
query55	82	74	71	71
query56	180	159	158	158
query57	1007	891	909	891
query58	185	152	162	152
query59	2744	2594	2425	2425
query60	207	182	178	178
query61	93	85	83	83
query62	617	374	359	359
query63	278	269	260	260
query64	5976	1797	1748	1748
query65	3320	3249	3225	3225
query66	1462	329	322	322
query67	14416	14138	14153	14138
query68	3768	485	520	485
query69	436	313	298	298
query70	1529	1484	1507	1484
query71	268	226	214	214
query72	3746	2850	2834	2834
query73	702	310	310	310
query74	6640	6278	6199	6199
query75	2840	2287	2280	2280
query76	2588	1032	1102	1032
query77	327	234	224	224
query78	9134	8946	8505	8505
query79	2804	484	474	474
query80	2009	327	314	314
query81	517	200	206	200
query82	695	90	81	81
query83	244	126	120	120
query84	289	77	73	73
query85	2244	343	336	336
query86	518	380	414	380
query87	3436	3279	3288	3279
query88	3777	2175	2168	2168
query89	422	350	364	350
query90	1921	191	184	184
query91	146	118	117	117
query92	58	43	44	43
query93	2680	437	440	437
query94	1350	161	163	161
query95	510	462	454	454
query96	634	314	314	314
query97	4254	4119	4109	4109
query98	213	195	184	184
query99	1060	704	682	682
Total cold run time: 281723 ms
Total hot run time: 172995 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee, data reload: false

query1	0.04	0.04	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.51	0.52
query6	1.20	0.64	0.66
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.55	0.49	0.51
query10	0.55	0.57	0.56
query11	0.12	0.08	0.09
query12	0.11	0.08	0.09
query13	0.61	0.61	0.62
query14	0.79	0.79	0.80
query15	0.80	0.78	0.75
query16	0.39	0.39	0.38
query17	1.02	1.00	0.96
query18	0.20	0.25	0.23
query19	1.94	1.77	1.79
query20	0.02	0.01	0.01
query21	15.42	0.57	0.58
query22	2.62	2.57	2.80
query23	17.35	0.76	0.75
query24	2.25	1.28	1.10
query25	0.41	0.29	0.15
query26	0.47	0.14	0.14
query27	0.05	0.05	0.05
query28	11.64	0.86	0.81
query29	12.50	3.17	3.14
query30	0.69	0.53	0.56
query31	2.79	0.35	0.35
query32	3.37	0.48	0.48
query33	3.23	3.24	3.23
query34	15.77	4.43	4.36
query35	4.33	4.32	4.25
query36	1.12	1.06	1.07
query37	0.07	0.05	0.05
query38	0.05	0.03	0.02
query39	0.03	0.01	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.32 s
Total hot run time: 31.9 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 634bb0b9e419ecca0007c550fa3bcfbcbfffa2ee with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

@jackwener jackwener self-requested a review January 27, 2024 09:52
@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37524 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false

------ Round 1 ----------------------------------
q1	17679	5801	4857	4857
q2	2037	140	130	130
q3	10712	923	940	923
q4	4698	731	745	731
q5	7698	2956	2905	2905
q6	186	118	117	117
q7	1143	722	705	705
q8	9438	2038	2005	2005
q9	7210	6344	6271	6271
q10	8099	2434	2417	2417
q11	409	198	208	198
q12	795	264	265	264
q13	18001	3310	3310	3310
q14	269	242	239	239
q15	525	493	485	485
q16	454	424	437	424
q17	921	453	489	453
q18	6957	6162	6111	6111
q19	4952	1389	1363	1363
q20	649	329	342	329
q21	6648	3083	2980	2980
q22	813	321	307	307
Total cold run time: 110293 ms
Total hot run time: 37524 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4999	5113	4942	4942
q2	359	230	231	230
q3	3012	2688	2701	2688
q4	1851	1558	1565	1558
q5	5355	5321	5310	5310
q6	203	116	113	113
q7	2135	1798	1780	1780
q8	3118	3182	3266	3182
q9	8664	8686	14759	8686
q10	5507	3464	3548	3464
q11	567	483	480	480
q12	721	592	623	592
q13	16926	3135	3049	3049
q14	281	249	259	249
q15	537	492	492	492
q16	531	474	495	474
q17	1947	1787	1696	1696
q18	9115	9958	8504	8504
q19	24130	1547	1539	1539
q20	4615	1906	1899	1899
q21	16053	5014	4889	4889
q22	1043	499	466	466
Total cold run time: 111669 ms
Total hot run time: 56282 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174285 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false

query1	936	339	332	332
query2	6537	2070	1936	1936
query3	6921	215	203	203
query4	32816	22528	22479	22479
query5	4597	383	458	383
query6	549	175	170	170
query7	5860	292	287	287
query8	515	190	186	186
query9	10473	2304	2280	2280
query10	421	215	206	206
query11	17946	15434	15366	15366
query12	122	71	68	68
query13	1678	399	379	379
query14	9255	6540	6508	6508
query15	213	182	180	180
query16	5939	256	255	255
query17	1410	473	477	473
query18	1795	261	257	257
query19	187	141	131	131
query20	74	70	64	64
query21	206	134	143	134
query22	4775	4823	4628	4628
query23	31685	30318	30450	30318
query24	12854	2833	2805	2805
query25	596	317	307	307
query26	1979	141	135	135
query27	3569	284	286	284
query28	7629	1827	1807	1807
query29	2243	631	639	631
query30	290	134	144	134
query31	911	700	725	700
query32	76	51	50	50
query33	704	203	213	203
query34	1230	461	471	461
query35	871	754	744	744
query36	1347	1237	1187	1187
query37	177	58	62	58
query38	3297	3179	3177	3177
query39	1323	1268	1272	1268
query40	336	86	84	84
query41	38	35	34	34
query42	86	78	87	78
query43	546	470	511	470
query44	1163	684	690	684
query45	196	176	177	176
query46	1046	665	657	657
query47	1588	1527	1505	1505
query48	397	306	305	305
query49	1276	298	290	290
query50	712	312	318	312
query51	5301	5186	5134	5134
query52	90	83	77	77
query53	324	271	264	264
query54	257	190	193	190
query55	80	75	76	75
query56	173	161	161	161
query57	981	907	910	907
query58	186	166	157	157
query59	2948	2746	2742	2742
query60	211	182	174	174
query61	82	86	92	86
query62	612	345	370	345
query63	292	259	268	259
query64	5785	1797	1738	1738
query65	3312	3239	3217	3217
query66	1460	339	311	311
query67	14539	14255	14127	14127
query68	4433	480	491	480
query69	426	303	286	286
query70	1546	1476	1471	1471
query71	298	218	227	218
query72	3960	2885	2878	2878
query73	692	306	318	306
query74	6765	6282	6264	6264
query75	2873	2262	2284	2262
query76	3282	1037	992	992
query77	367	244	237	237
query78	9099	8836	8736	8736
query79	2539	505	482	482
query80	1987	327	318	318
query81	519	198	197	197
query82	742	82	86	82
query83	246	120	120	120
query84	285	75	74	74
query85	2393	343	336	336
query86	517	401	409	401
query87	3495	3247	3295	3247
query88	3881	2157	2183	2157
query89	433	352	360	352
query90	1935	188	184	184
query91	152	117	119	117
query92	54	43	42	42
query93	2433	429	407	407
query94	1397	165	162	162
query95	501	457	454	454
query96	618	311	307	307
query97	4224	4136	4137	4136
query98	205	201	190	190
query99	1058	662	657	657
Total cold run time: 291378 ms
Total hot run time: 174285 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.05	0.06
query4	1.67	0.10	0.10
query5	0.53	0.51	0.51
query6	1.19	0.63	0.61
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.56	0.52	0.49
query10	0.57	0.56	0.57
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.61	0.61	0.61
query14	0.79	0.80	0.80
query15	0.79	0.79	0.79
query16	0.38	0.39	0.40
query17	0.96	1.03	1.03
query18	0.23	0.23	0.25
query19	1.83	1.80	1.79
query20	0.01	0.00	0.01
query21	15.40	0.57	0.58
query22	2.46	2.12	1.74
query23	17.64	0.89	0.70
query24	3.00	0.41	0.84
query25	0.29	0.14	0.14
query26	0.55	0.14	0.14
query27	0.05	0.06	0.05
query28	11.50	0.85	0.86
query29	12.48	3.17	3.27
query30	0.65	0.56	0.53
query31	2.78	0.36	0.35
query32	3.34	0.50	0.49
query33	3.21	3.22	3.24
query34	15.75	4.29	4.34
query35	4.47	4.35	4.41
query36	1.11	1.07	1.05
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.03	0.02	0.01
query40	0.16	0.12	0.13
query41	0.07	0.01	0.02
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.82 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9a12a38cab462404742ab49aab1eaeeff3255fb1 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38016 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false

------ Round 1 ----------------------------------
q1	18015	5162	5172	5162
q2	2706	143	136	136
q3	11235	957	931	931
q4	4923	727	733	727
q5	8497	2920	2827	2827
q6	194	122	122	122
q7	1168	759	746	746
q8	9425	2008	2023	2008
q9	7219	6339	6335	6335
q10	8125	2379	2425	2379
q11	420	207	214	207
q12	796	285	284	284
q13	18018	3296	3286	3286
q14	280	240	249	240
q15	528	494	475	475
q16	479	422	403	403
q17	939	559	520	520
q18	7069	6070	6130	6070
q19	1579	1384	1432	1384
q20	586	349	361	349
q21	7155	3145	3125	3125
q22	812	304	300	300
Total cold run time: 110168 ms
Total hot run time: 38016 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5028	4910	4922	4910
q2	360	245	236	236
q3	3027	2848	2772	2772
q4	1802	1601	1491	1491
q5	5323	5361	5131	5131
q6	203	120	115	115
q7	2171	1843	1802	1802
q8	3115	3157	3235	3157
q9	8668	8617	11278	8617
q10	5683	3409	3449	3409
q11	544	482	481	481
q12	757	602	656	602
q13	16952	3164	3101	3101
q14	273	269	259	259
q15	541	490	496	490
q16	514	463	463	463
q17	2030	1638	1728	1638
q18	9117	11344	8874	8874
q19	23910	1516	1502	1502
q20	4589	1909	1895	1895
q21	15727	4828	4784	4784
q22	1211	478	488	478
Total cold run time: 111545 ms
Total hot run time: 56207 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173626 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false

query1	940	335	333	333
query2	6550	1964	1927	1927
query3	6689	201	194	194
query4	32465	22107	22113	22107
query5	4452	420	424	420
query6	252	165	173	165
query7	4609	277	274	274
query8	243	182	193	182
query9	8580	2279	2258	2258
query10	402	210	199	199
query11	18042	15436	15473	15436
query12	125	70	66	66
query13	1674	365	374	365
query14	9236	6533	6452	6452
query15	208	184	190	184
query16	5730	253	244	244
query17	954	466	480	466
query18	1807	276	254	254
query19	183	141	133	133
query20	77	77	69	69
query21	201	131	131	131
query22	4575	4695	4644	4644
query23	31306	30375	30410	30375
query24	12192	2809	2825	2809
query25	597	315	307	307
query26	1817	150	144	144
query27	3185	288	278	278
query28	7657	1827	1815	1815
query29	1865	625	594	594
query30	283	140	138	138
query31	912	716	724	716
query32	79	47	49	47
query33	696	219	205	205
query34	1181	469	463	463
query35	864	791	763	763
query36	1375	1239	1167	1167
query37	94	58	57	57
query38	3283	3176	3139	3139
query39	1317	1250	1242	1242
query40	342	97	84	84
query41	39	35	35	35
query42	88	81	83	81
query43	527	491	515	491
query44	1142	690	689	689
query45	192	178	174	174
query46	1027	659	651	651
query47	1584	1542	1493	1493
query48	381	320	311	311
query49	1170	291	292	291
query50	662	316	316	316
query51	5310	5217	5194	5194
query52	99	78	79	78
query53	316	261	263	261
query54	260	198	191	191
query55	82	80	74	74
query56	173	163	165	163
query57	998	906	889	889
query58	182	160	156	156
query59	2857	2759	2705	2705
query60	208	186	182	182
query61	83	82	82	82
query62	659	357	347	347
query63	279	259	257	257
query64	5964	1771	1740	1740
query65	3303	3249	3214	3214
query66	1242	319	306	306
query67	14589	14250	14303	14250
query68	3722	504	503	503
query69	445	296	322	296
query70	1559	1509	1535	1509
query71	287	205	214	205
query72	3756	2817	2854	2817
query73	694	327	303	303
query74	6628	6233	6193	6193
query75	2847	2256	2316	2256
query76	2600	1061	958	958
query77	378	240	239	239
query78	9141	8779	8450	8450
query79	2834	491	494	491
query80	1996	327	328	327
query81	516	199	202	199
query82	722	87	84	84
query83	230	123	119	119
query84	294	83	72	72
query85	2207	339	342	339
query86	510	405	369	369
query87	3488	3261	3248	3248
query88	3739	2156	2155	2155
query89	393	373	361	361
query90	1933	189	185	185
query91	151	120	118	118
query92	54	41	45	41
query93	2855	417	442	417
query94	1361	162	162	162
query95	509	468	460	460
query96	617	320	326	320
query97	4241	4146	4112	4112
query98	218	194	189	189
query99	1065	673	662	662
Total cold run time: 282445 ms
Total hot run time: 173626 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.06	0.05
query4	1.65	0.10	0.10
query5	0.53	0.51	0.51
query6	1.19	0.63	0.61
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.53	0.51	0.50
query10	0.55	0.57	0.57
query11	0.11	0.08	0.09
query12	0.11	0.09	0.10
query13	0.62	0.60	0.60
query14	0.80	0.80	0.80
query15	0.79	0.76	0.77
query16	0.39	0.39	0.40
query17	1.02	0.99	1.03
query18	0.22	0.25	0.25
query19	1.86	1.70	1.74
query20	0.01	0.02	0.02
query21	15.40	0.58	0.57
query22	2.27	2.52	1.52
query23	17.38	1.08	0.78
query24	2.51	0.56	1.43
query25	0.29	0.18	0.17
query26	0.53	0.14	0.13
query27	0.04	0.04	0.04
query28	10.86	0.86	0.86
query29	12.59	3.22	3.25
query30	0.67	0.56	0.53
query31	2.78	0.35	0.36
query32	3.34	0.48	0.48
query33	3.24	3.19	3.25
query34	15.75	4.37	4.36
query35	4.31	4.29	4.29
query36	1.11	1.06	1.07
query37	0.07	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.26 s
Total hot run time: 30.47 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3a46476f6c1218e83afc0abd89dd9fc5d04b5a13 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37235 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false

------ Round 1 ----------------------------------
q1	17729	4773	4994	4773
q2	2059	154	136	136
q3	10686	925	912	912
q4	4727	728	684	684
q5	7699	2818	2758	2758
q6	184	120	120	120
q7	1150	712	702	702
q8	9376	2016	2008	2008
q9	7266	6346	6293	6293
q10	8105	2384	2432	2384
q11	417	198	200	198
q12	749	286	291	286
q13	17998	3305	3272	3272
q14	279	245	249	245
q15	525	498	495	495
q16	456	419	415	415
q17	926	550	495	495
q18	6978	5996	5987	5987
q19	2223	1388	1369	1369
q20	625	345	337	337
q21	6801	3103	3082	3082
q22	789	284	301	284
Total cold run time: 107747 ms
Total hot run time: 37235 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5210	4982	5015	4982
q2	352	244	243	243
q3	3022	2711	2659	2659
q4	1837	1581	1534	1534
q5	5216	5222	5265	5222
q6	200	117	115	115
q7	2148	1728	1759	1728
q8	3098	3250	3262	3250
q9	8644	8913	18997	8913
q10	5638	3392	3545	3392
q11	566	469	473	469
q12	759	614	613	613
q13	16961	3073	3057	3057
q14	277	251	256	251
q15	538	507	484	484
q16	513	472	446	446
q17	1952	1771	1762	1762
q18	8861	8506	11537	8506
q19	23981	1585	1551	1551
q20	4567	1933	1881	1881
q21	14780	4900	4767	4767
q22	1048	524	465	465
Total cold run time: 110168 ms
Total hot run time: 56290 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173037 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false

query1	947	336	322	322
query2	6558	1947	2012	1947
query3	6694	202	195	195
query4	33558	22020	22050	22020
query5	4465	420	346	346
query6	252	151	154	151
query7	4600	255	253	253
query8	238	193	177	177
query9	8550	2265	2247	2247
query10	408	203	204	203
query11	17873	15426	15340	15340
query12	120	68	67	67
query13	1697	386	371	371
query14	9216	6438	6408	6408
query15	209	175	186	175
query16	5695	260	245	245
query17	951	472	476	472
query18	1816	261	249	249
query19	178	132	132	132
query20	70	68	73	68
query21	201	131	130	130
query22	5009	4473	4734	4473
query23	31373	30339	30379	30339
query24	7412	2765	2731	2731
query25	423	297	304	297
query26	796	135	132	132
query27	2773	276	279	276
query28	3997	1836	1810	1810
query29	895	614	610	610
query30	215	137	133	133
query31	891	702	719	702
query32	62	52	48	48
query33	418	210	210	210
query34	805	454	460	454
query35	841	758	723	723
query36	1296	1180	1180	1180
query37	94	60	57	57
query38	3289	3180	3132	3132
query39	1316	1263	1260	1260
query40	198	98	84	84
query41	37	34	34	34
query42	88	83	87	83
query43	544	496	472	472
query44	1063	677	674	674
query45	187	177	175	175
query46	1025	640	642	640
query47	1578	1556	1478	1478
query48	379	302	302	302
query49	603	280	281	280
query50	694	301	318	301
query51	5322	5217	5219	5217
query52	83	80	82	80
query53	313	263	262	262
query54	225	188	186	186
query55	79	77	70	70
query56	178	160	169	160
query57	986	905	936	905
query58	178	155	162	155
query59	2718	2714	2689	2689
query60	195	178	179	178
query61	83	86	81	81
query62	434	351	364	351
query63	293	261	267	261
query64	2541	1786	1712	1712
query65	3307	3253	3201	3201
query66	1127	310	321	310
query67	14504	14438	14338	14338
query68	3793	482	486	482
query69	446	302	293	293
query70	1504	1492	1483	1483
query71	267	207	214	207
query72	3716	2842	2831	2831
query73	684	310	306	306
query74	6610	6252	6356	6252
query75	2748	2263	2296	2263
query76	2666	1023	998	998
query77	381	231	233	231
query78	9013	8959	8571	8571
query79	2385	481	476	476
query80	2020	330	310	310
query81	495	199	200	199
query82	670	83	81	81
query83	275	119	111	111
query84	282	75	71	71
query85	1699	333	339	333
query86	518	402	372	372
query87	3425	3278	3267	3267
query88	3880	2156	2147	2147
query89	425	357	345	345
query90	2005	191	180	180
query91	150	116	119	116
query92	52	45	45	45
query93	2380	426	406	406
query94	1388	162	158	158
query95	499	461	453	453
query96	627	304	300	300
query97	4267	4121	4116	4116
query98	220	186	184	184
query99	1052	686	700	686
Total cold run time: 265705 ms
Total hot run time: 173037 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad, data reload: false

query1	0.04	0.04	0.03
query2	0.05	0.02	0.02
query3	0.24	0.06	0.05
query4	1.69	0.10	0.10
query5	0.53	0.51	0.52
query6	1.18	0.63	0.63
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.56	0.51	0.50
query10	0.56	0.55	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.59	0.60	0.61
query14	0.79	0.82	0.83
query15	0.82	0.77	0.78
query16	0.36	0.39	0.38
query17	1.08	1.05	0.98
query18	0.20	0.26	0.28
query19	1.84	1.79	1.65
query20	0.02	0.01	0.01
query21	15.35	0.59	0.56
query22	2.51	2.25	2.14
query23	17.18	0.78	0.81
query24	2.33	1.29	0.65
query25	0.29	0.12	0.19
query26	0.52	0.14	0.13
query27	0.05	0.04	0.03
query28	11.87	0.85	0.88
query29	12.56	3.16	3.10
query30	0.65	0.56	0.59
query31	2.78	0.36	0.35
query32	3.35	0.48	0.48
query33	3.23	3.23	3.23
query34	15.76	4.27	4.22
query35	4.34	4.30	4.27
query36	1.10	1.07	1.06
query37	0.06	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.1 s
Total hot run time: 30.87 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1cd3149d48700b6ad608ec2e72c69d716dcd7fad with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37710 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false

------ Round 1 ----------------------------------
q1	17817	4959	5104	4959
q2	2806	146	143	143
q3	11530	942	931	931
q4	5108	827	705	705
q5	7877	2944	2769	2769
q6	190	124	124	124
q7	1195	742	738	738
q8	9486	1992	2033	1992
q9	7232	6339	6338	6338
q10	8049	2441	2416	2416
q11	410	204	189	189
q12	755	278	279	278
q13	18015	3348	3298	3298
q14	270	248	251	248
q15	529	492	491	491
q16	462	398	398	398
q17	949	579	496	496
q18	6953	6180	6227	6180
q19	1573	1349	1425	1349
q20	584	359	343	343
q21	6630	3171	3029	3029
q22	810	296	301	296
Total cold run time: 109230 ms
Total hot run time: 37710 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5166	4849	4850	4849
q2	342	225	236	225
q3	3011	2773	2742	2742
q4	1843	1603	1570	1570
q5	5326	5315	5235	5235
q6	212	116	115	115
q7	2188	1826	1823	1823
q8	3142	3196	3237	3196
q9	8575	8535	14087	8535
q10	5477	3488	3473	3473
q11	546	474	465	465
q12	770	604	605	604
q13	16937	3059	3050	3050
q14	277	256	268	256
q15	532	501	493	493
q16	509	455	469	455
q17	2059	1651	1716	1651
q18	8825	11628	8792	8792
q19	25184	1545	1493	1493
q20	4593	1908	1880	1880
q21	16248	4746	4794	4746
q22	1603	461	496	461
Total cold run time: 113365 ms
Total hot run time: 56109 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172655 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false

query1	927	327	318	318
query2	6566	1903	1928	1903
query3	6703	198	198	198
query4	33188	22048	21995	21995
query5	4453	352	355	352
query6	247	152	149	149
query7	4611	260	258	258
query8	247	173	173	173
query9	8599	2270	2267	2267
query10	408	225	206	206
query11	17622	15390	15400	15390
query12	118	69	65	65
query13	1672	376	376	376
query14	9210	6562	6428	6428
query15	204	183	179	179
query16	5701	255	251	251
query17	948	474	488	474
query18	1804	253	254	253
query19	172	136	131	131
query20	76	64	67	64
query21	193	134	131	131
query22	4880	4747	4569	4569
query23	31228	30349	30419	30349
query24	12695	2749	2790	2749
query25	608	308	308	308
query26	1970	140	140	140
query27	3460	282	285	282
query28	7715	1818	1827	1818
query29	2308	624	618	618
query30	286	133	142	133
query31	896	710	732	710
query32	76	57	46	46
query33	702	204	205	204
query34	1300	456	471	456
query35	863	752	748	748
query36	1282	1198	1169	1169
query37	175	59	55	55
query38	3242	3143	3151	3143
query39	1313	1248	1241	1241
query40	344	87	80	80
query41	37	35	34	34
query42	85	81	83	81
query43	493	472	469	469
query44	1130	700	690	690
query45	187	177	173	173
query46	1041	647	646	646
query47	1606	1548	1520	1520
query48	374	305	294	294
query49	1226	279	286	279
query50	674	311	307	307
query51	5277	5149	5131	5131
query52	85	79	72	72
query53	318	263	270	263
query54	264	180	187	180
query55	76	77	75	75
query56	169	154	164	154
query57	983	893	915	893
query58	171	147	159	147
query59	2651	2518	2566	2518
query60	200	175	175	175
query61	86	85	84	84
query62	631	362	354	354
query63	270	260	266	260
query64	5967	1749	1736	1736
query65	3277	3223	3188	3188
query66	1465	316	310	310
query67	14330	14140	14154	14140
query68	3666	481	490	481
query69	427	296	306	296
query70	1513	1423	1477	1423
query71	288	214	206	206
query72	3770	2852	2876	2852
query73	693	307	307	307
query74	6667	6170	6123	6123
query75	2736	2311	2289	2289
query76	2682	1037	982	982
query77	434	233	225	225
query78	9031	8621	8544	8544
query79	3116	489	498	489
query80	1916	314	316	314
query81	525	195	197	195
query82	1117	78	84	78
query83	394	116	115	115
query84	284	78	70	70
query85	1767	318	321	318
query86	497	387	383	383
query87	3397	3264	3259	3259
query88	3774	2159	2148	2148
query89	404	337	374	337
query90	2091	183	185	183
query91	145	119	116	116
query92	57	44	42	42
query93	2632	414	417	414
query94	1516	158	158	158
query95	501	465	449	449
query96	634	314	314	314
query97	4288	4133	4163	4133
query98	221	193	189	189
query99	1132	671	688	671
Total cold run time: 284400 ms
Total hot run time: 172655 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987, data reload: false

query1	0.04	0.02	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.69	0.10	0.09
query5	0.53	0.52	0.52
query6	1.27	0.64	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.50	0.48
query10	0.54	0.54	0.54
query11	0.11	0.09	0.08
query12	0.11	0.08	0.09
query13	0.60	0.60	0.61
query14	0.78	0.81	0.80
query15	0.79	0.78	0.78
query16	0.38	0.39	0.38
query17	1.02	1.04	1.04
query18	0.22	0.24	0.24
query19	1.85	1.70	1.79
query20	0.02	0.01	0.01
query21	15.40	0.58	0.57
query22	2.54	2.46	2.10
query23	17.38	0.76	0.80
query24	2.84	0.88	0.45
query25	0.34	0.25	0.05
query26	0.24	0.13	0.14
query27	0.05	0.05	0.04
query28	12.36	0.87	0.83
query29	12.56	3.06	3.10
query30	0.65	0.53	0.56
query31	2.81	0.35	0.35
query32	3.35	0.47	0.48
query33	3.21	3.21	3.20
query34	15.66	4.33	4.38
query35	4.29	4.28	4.37
query36	1.12	1.07	1.07
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.12	0.16
query41	0.08	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.01	0.02
Total cold run time: 106.06 s
Total hot run time: 30.55 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit faf03986fa7fe0caed521d9a668d53ca55dc6987 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.1 seconds inserted 10000000 Rows, about 709K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36532 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false

------ Round 1 ----------------------------------
q1	17684	4443	4418	4418
q2	2039	139	127	127
q3	10601	897	879	879
q4	4649	753	702	702
q5	7663	2693	2847	2693
q6	183	119	119	119
q7	1113	729	707	707
q8	9270	1954	1982	1954
q9	7201	6313	6290	6290
q10	8069	2404	2450	2404
q11	415	201	192	192
q12	743	279	278	278
q13	18019	3283	3313	3283
q14	278	246	253	246
q15	529	488	482	482
q16	463	415	392	392
q17	931	509	506	506
q18	6828	5900	5880	5880
q19	1561	1456	1355	1355
q20	579	309	330	309
q21	6962	3009	3032	3009
q22	806	311	307	307
Total cold run time: 106586 ms
Total hot run time: 36532 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4459	4429	4370	4370
q2	332	237	241	237
q3	2994	2838	2819	2819
q4	1865	1667	1677	1667
q5	5180	5266	5234	5234
q6	188	112	115	112
q7	2181	1786	1795	1786
q8	3104	3243	3241	3241
q9	8399	8210	8213	8210
q10	5801	3600	3522	3522
q11	550	474	455	455
q12	728	591	572	572
q13	13573	3076	3080	3076
q14	281	264	259	259
q15	532	495	489	489
q16	522	474	467	467
q17	1829	1714	1672	1672
q18	7987	7704	7498	7498
q19	8805	1554	1543	1543
q20	2153	1909	1901	1901
q21	7143	4735	4772	4735
q22	513	471	483	471
Total cold run time: 79119 ms
Total hot run time: 54336 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172409 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false

query1	934	344	325	325
query2	6565	1894	1821	1821
query3	6696	201	191	191
query4	27335	21987	22140	21987
query5	4459	355	343	343
query6	260	162	169	162
query7	4604	262	253	253
query8	253	172	190	172
query9	9145	2278	2253	2253
query10	416	224	196	196
query11	16859	15373	15497	15373
query12	128	66	64	64
query13	1682	380	371	371
query14	9380	6477	6424	6424
query15	225	183	188	183
query16	6472	255	247	247
query17	1902	462	478	462
query18	1909	251	245	245
query19	178	134	129	129
query20	70	73	68	68
query21	193	123	130	123
query22	4995	4734	4758	4734
query23	31070	30268	30178	30178
query24	8814	2686	2762	2686
query25	538	313	312	312
query26	826	137	137	137
query27	2748	281	295	281
query28	5898	1826	1824	1824
query29	928	622	610	610
query30	277	138	139	138
query31	917	706	700	700
query32	88	55	52	52
query33	501	203	202	202
query34	825	435	453	435
query35	860	756	759	756
query36	1229	1186	1138	1138
query37	93	55	62	55
query38	3260	3137	3123	3123
query39	1313	1253	1250	1250
query40	189	87	84	84
query41	37	35	34	34
query42	83	78	80	78
query43	510	481	491	481
query44	1011	682	687	682
query45	191	180	176	176
query46	1026	641	628	628
query47	1608	1528	1529	1528
query48	380	314	297	297
query49	1117	285	278	278
query50	681	302	308	302
query51	5320	5112	5136	5112
query52	95	83	82	82
query53	322	255	257	255
query54	230	194	179	179
query55	82	75	80	75
query56	170	162	160	160
query57	995	921	909	909
query58	173	152	158	152
query59	2632	2446	2299	2299
query60	194	174	173	173
query61	86	87	86	86
query62	571	368	347	347
query63	277	255	261	255
query64	4042	1793	1740	1740
query65	3301	3234	3213	3213
query66	1136	326	326	326
query67	14438	14354	14096	14096
query68	4037	499	509	499
query69	441	311	312	311
query70	1451	1517	1489	1489
query71	280	212	209	209
query72	3729	2867	2889	2867
query73	686	309	309	309
query74	6578	6307	6300	6300
query75	2837	2343	2267	2267
query76	2975	972	927	927
query77	378	233	225	225
query78	9172	8772	8558	8558
query79	2658	493	498	493
query80	2075	334	333	333
query81	528	196	197	196
query82	719	84	81	81
query83	251	119	115	115
query84	277	77	72	72
query85	1947	356	323	323
query86	506	380	385	380
query87	3439	3274	3324	3274
query88	3912	2179	2170	2170
query89	413	364	338	338
query90	1905	187	185	185
query91	153	117	118	117
query92	51	46	44	44
query93	2507	422	424	422
query94	1260	157	161	157
query95	508	467	449	449
query96	624	334	315	315
query97	4237	4115	4124	4115
query98	202	197	189	189
query99	1005	694	650	650
Total cold run time: 267483 ms
Total hot run time: 172409 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.23	0.07	0.06
query4	1.66	0.10	0.10
query5	0.53	0.52	0.51
query6	1.21	0.64	0.64
query7	0.01	0.01	0.00
query8	0.04	0.02	0.02
query9	0.57	0.51	0.50
query10	0.57	0.56	0.55
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.61	0.61	0.61
query14	0.80	0.80	0.80
query15	0.81	0.79	0.78
query16	0.37	0.38	0.39
query17	1.04	1.01	1.04
query18	0.22	0.28	0.22
query19	1.83	1.70	1.72
query20	0.01	0.01	0.01
query21	15.41	0.59	0.58
query22	2.71	2.54	2.00
query23	17.42	0.86	0.86
query24	2.31	0.92	0.73
query25	0.29	0.12	0.14
query26	0.48	0.15	0.14
query27	0.05	0.05	0.06
query28	12.19	0.84	0.83
query29	12.57	3.20	3.21
query30	0.57	0.50	0.50
query31	2.79	0.34	0.35
query32	3.36	0.49	0.48
query33	3.25	3.20	3.22
query34	15.82	4.34	4.32
query35	4.35	4.28	4.33
query36	1.14	1.07	1.06
query37	0.07	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.96 s
Total hot run time: 31.06 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b27e6aed4967a20d0c53b6eafd4bee423d8f02b5 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

Copy link
Contributor

@starocean999 starocean999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@jackwener jackwener merged commit be44013 into apache:master Feb 1, 2024
27 of 28 checks passed
@iwanttobepowerful iwanttobepowerful deleted the fix30264 branch February 1, 2024 10:00
yiguolei pushed a commit that referenced this pull request Feb 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged p0_w reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Nereids planner output a wrong extra column
5 participants