Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids) fix a visitor bug in CommonSubExpressionOpt #33154

Merged
merged 2 commits into from
Apr 11, 2024
Merged

[fix](nereids) fix a visitor bug in CommonSubExpressionOpt #33154

merged 2 commits into from
Apr 11, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented Apr 2, 2024 •
edited

Proposed changes

this pr contains 2 fix:

  1. cse visitor does not visit project's child
  2. variant data type column should be skipped. refer to regression case: variant_p0/tpch/sql/q16_trans.sql

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

englefly commented Apr 2, 2024

run buildall

morrySnow
morrySnow previously approved these changes Apr 2, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39186 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 52c598b21733cfdf94e36539b17003d5d48131dd, data reload: false

------ Round 1 ----------------------------------
q1	17624	4082	4047	4047
q2	2007	186	176	176
q3	10488	1265	1414	1265
q4	10215	889	1052	889
q5	7963	2974	2936	2936
q6	217	133	130	130
q7	1103	639	603	603
q8	9397	2053	2020	2020
q9	6689	6230	6174	6174
q10	8459	3545	3499	3499
q11	416	237	239	237
q12	383	214	215	214
q13	19454	2916	2929	2916
q14	278	241	250	241
q15	536	503	494	494
q16	503	395	378	378
q17	940	889	919	889
q18	8084	6945	6926	6926
q19	3310	1520	1541	1520
q20	592	316	320	316
q21	3526	3005	3138	3005
q22	370	311	322	311
Total cold run time: 112554 ms
Total hot run time: 39186 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4117	4079	4025	4025
q2	326	225	237	225
q3	2984	2942	2923	2923
q4	1841	1826	1803	1803
q5	5213	5186	5218	5186
q6	204	124	125	124
q7	2210	1831	1802	1802
q8	3183	3260	3337	3260
q9	8466	8447	8472	8447
q10	3963	3981	4046	3981
q11	590	495	466	466
q12	742	581	600	581
q13	16850	3181	3108	3108
q14	310	287	273	273
q15	527	474	482	474
q16	490	443	415	415
q17	1751	1718	1733	1718
q18	8122	7616	7679	7616
q19	3060	1714	1641	1641
q20	1920	1807	1808	1807
q21	5251	4944	4937	4937
q22	511	460	458	458
Total cold run time: 72631 ms
Total hot run time: 55270 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183121 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 52c598b21733cfdf94e36539b17003d5d48131dd, data reload: false

query1	892	1130	363	363
query2	6187	2033	2030	2030
query3	6660	211	208	208
query4	25019	21856	21717	21717
query5	4171	410	409	409
query6	281	202	192	192
query7	4604	323	310	310
query8	252	179	175	175
query9	8454	2266	2284	2266
query10	431	260	268	260
query11	15101	14512	14520	14512
query12	135	96	94	94
query13	1623	371	383	371
query14	8454	7078	6920	6920
query15	218	178	176	176
query16	6738	284	281	281
query17	971	613	605	605
query18	1870	304	290	290
query19	218	165	163	163
query20	99	97	99	97
query21	199	141	136	136
query22	4969	4799	4828	4799
query23	33412	32426	32801	32426
query24	12598	3348	3217	3217
query25	708	457	414	414
query26	1725	182	174	174
query27	3177	402	425	402
query28	6880	1908	1891	1891
query29	1210	638	614	614
query30	326	188	183	183
query31	1022	779	750	750
query32	101	66	65	65
query33	714	257	255	255
query34	1094	551	533	533
query35	891	753	754	753
query36	1024	903	889	889
query37	157	88	91	88
query38	3744	3579	3663	3579
query39	1637	1626	1600	1600
query40	258	143	147	143
query41	52	46	49	46
query42	119	107	113	107
query43	447	410	405	405
query44	1135	734	737	734
query45	313	273	273	273
query46	1115	854	811	811
query47	2003	1919	1905	1905
query48	414	315	312	312
query49	936	387	370	370
query50	826	426	430	426
query51	7010	6857	6912	6857
query52	115	103	99	99
query53	386	304	311	304
query54	331	246	250	246
query55	97	82	84	82
query56	259	247	239	239
query57	1280	1152	1166	1152
query58	252	233	234	233
query59	2661	2499	2525	2499
query60	260	229	231	229
query61	88	88	86	86
query62	654	472	469	469
query63	312	284	287	284
query64	5781	3053	3131	3053
query65	3064	3017	3027	3017
query66	1309	315	313	313
query67	15470	14771	15112	14771
query68	9496	575	610	575
query69	593	343	327	327
query70	1417	1112	1104	1104
query71	504	279	275	275
query72	6795	2597	2426	2426
query73	1580	334	327	327
query74	6709	6262	6274	6262
query75	3597	2346	2290	2290
query76	5569	1101	1211	1101
query77	579	246	249	246
query78	11196	10274	10283	10274
query79	11883	533	534	533
query80	1826	439	422	422
query81	524	261	250	250
query82	418	105	106	105
query83	197	160	158	158
query84	267	89	88	88
query85	958	292	282	282
query86	370	268	276	268
query87	3726	3489	3516	3489
query88	4388	2347	2368	2347
query89	536	375	370	370
query90	1920	175	183	175
query91	130	108	104	104
query92	69	55	55	55
query93	6987	525	546	525
query94	1191	195	191	191
query95	1092	1093	1103	1093
query96	638	287	277	277
query97	2675	2462	2496	2462
query98	233	210	223	210
query99	1173	842	838	838
Total cold run time: 304149 ms
Total hot run time: 183121 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 52c598b21733cfdf94e36539b17003d5d48131dd, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.66	0.07	0.08
query5	0.49	0.49	0.49
query6	1.16	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.51	0.52
query10	0.56	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.59
query14	0.76	0.78	0.80
query15	0.87	0.83	0.82
query16	0.35	0.36	0.37
query17	1.00	0.99	1.00
query18	0.25	0.25	0.26
query19	1.86	1.76	1.72
query20	0.01	0.01	0.01
query21	15.40	0.75	0.74
query22	3.16	6.39	1.36
query23	17.51	1.21	1.09
query24	1.45	0.22	0.37
query25	0.12	0.09	0.07
query26	0.29	0.17	0.19
query27	0.08	0.08	0.08
query28	13.43	0.94	0.95
query29	12.50	3.35	3.33
query30	0.26	0.05	0.05
query31	2.87	0.39	0.39
query32	3.25	0.47	0.47
query33	2.88	2.87	2.88
query34	15.49	4.42	4.36
query35	4.38	4.39	4.39
query36	0.68	0.46	0.47
query37	0.18	0.18	0.15
query38	0.18	0.15	0.17
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 105.35 s
Total hot run time: 29.79 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 52c598b21733cfdf94e36539b17003d5d48131dd with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

starocean999
starocean999 previously approved these changes Apr 3, 2024
View reviewed changes
@englefly
Copy link
Contributor Author

englefly commented Apr 3, 2024

run p0

@englefly
Copy link
Contributor Author

englefly commented Apr 3, 2024

run feut

@englefly
Copy link
Contributor Author

englefly commented Apr 3, 2024

run p0

2 similar comments
@englefly
Copy link
Contributor Author

englefly commented Apr 3, 2024

run p0

@englefly
Copy link
Contributor Author

englefly commented Apr 4, 2024

run p0

@yiguolei
Copy link
Contributor

yiguolei commented Apr 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38437 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 925ddb1af44c81a12857bcb74a81bf1d867399e2, data reload: false

------ Round 1 ----------------------------------
q1	17625	4085	4050	4050
q2	2007	185	178	178
q3	10484	1192	1350	1192
q4	10204	841	965	841
q5	7467	2905	2910	2905
q6	216	135	134	134
q7	1104	612	619	612
q8	9394	2012	2016	2012
q9	6628	6213	6130	6130
q10	8445	3522	3503	3503
q11	423	239	243	239
q12	372	210	210	210
q13	17785	2905	2921	2905
q14	267	228	240	228
q15	525	485	483	483
q16	517	368	374	368
q17	950	888	865	865
q18	7255	6426	6343	6343
q19	1591	1532	1522	1522
q20	563	330	333	330
q21	3502	3122	3083	3083
q22	364	304	319	304
Total cold run time: 107688 ms
Total hot run time: 38437 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4021	4021	4016	4016
q2	322	221	223	221
q3	2958	2934	2982	2934
q4	1865	1851	1882	1851
q5	5230	5215	5235	5215
q6	204	125	124	124
q7	2221	1780	1814	1780
q8	3205	3257	3254	3254
q9	8470	8433	8409	8409
q10	3736	3868	4034	3868
q11	562	448	451	448
q12	760	601	583	583
q13	16986	3118	3118	3118
q14	319	285	293	285
q15	534	493	492	492
q16	508	433	441	433
q17	1756	1725	1762	1725
q18	8137	7685	7719	7685
q19	1695	1686	1672	1672
q20	2033	1879	1815	1815
q21	5130	4946	4930	4930
q22	496	441	463	441
Total cold run time: 71148 ms
Total hot run time: 55299 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 925ddb1af44c81a12857bcb74a81bf1d867399e2, data reload: false

query1	1222	379	368	368
query2	7398	1938	1949	1938
query3	6674	218	215	215
query4	24296	21529	22061	21529
query5	4213	428	425	425
query6	297	210	197	197
query7	4774	316	305	305
query8	249	180	181	180
query9	10489	2216	2218	2216
query10	486	266	266	266
query11	15132	14625	14875	14625
query12	147	92	89	89
query13	1634	379	381	379
query14	9064	6902	6941	6902
query15	213	168	182	168
query16	7757	269	269	269
query17	1477	578	567	567
query18	1973	286	284	284
query19	192	144	149	144
query20	94	87	87	87
query21	191	128	130	128
query22	5002	4847	4747	4747
query23	33456	32509	32801	32509
query24	11212	3221	3198	3198
query25	650	385	379	379
query26	1758	156	163	156
query27	3174	379	387	379
query28	7128	1906	1883	1883
query29	1181	609	579	579
query30	306	170	187	170
query31	974	724	740	724
query32	96	63	53	53
query33	662	239	241	239
query34	1196	503	510	503
query35	845	717	728	717
query36	996	868	871	868
query37	224	73	76	73
query38	3641	3567	3562	3562
query39	1641	1632	1587	1587
query40	235	139	134	134
query41	51	45	45	45
query42	108	101	103	101
query43	456	413	412	412
query44	1190	772	733	733
query45	297	272	279	272
query46	1081	836	812	812
query47	1995	1901	1890	1890
query48	393	308	325	308
query49	973	377	384	377
query50	819	408	420	408
query51	7029	6871	6904	6871
query52	112	96	121	96
query53	354	290	289	289
query54	280	251	242	242
query55	84	76	77	76
query56	253	237	239	237
query57	1294	1167	1193	1167
query58	234	217	219	217
query59	2655	2511	2270	2270
query60	266	241	235	235
query61	112	108	108	108
query62	669	448	461	448
query63	308	275	281	275
query64	5556	3142	3136	3136
query65	3051	2991	3002	2991
query66	1234	313	306	306
query67	15467	14964	14709	14709
query68	9057	576	577	576
query69	533	293	295	293
query70	1357	1143	1113	1113
query71	540	265	276	265
query72	6131	2541	2402	2402
query73	857	324	321	321
query74	6885	6345	6424	6345
query75	3822	2254	2280	2254
query76	6213	1149	1232	1149
query77	917	254	267	254
query78	10755	10060	10109	10060
query79	10924	545	529	529
query80	1217	428	417	417
query81	489	233	232	232
query82	669	93	93	93
query83	204	167	165	165
query84	261	85	88	85
query85	1398	271	267	267
query86	381	294	306	294
query87	3639	3479	3495	3479
query88	3886	2359	2367	2359
query89	536	360	369	360
query90	2015	175	173	173
query91	131	104	101	101
query92	64	47	47	47
query93	6679	518	520	518
query94	1210	182	175	175
query95	414	314	307	307
query96	606	268	266	266
query97	2684	2479	2462	2462
query98	233	232	225	225
query99	1277	811	845	811
Total cold run time: 304654 ms
Total hot run time: 181169 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 925ddb1af44c81a12857bcb74a81bf1d867399e2, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.06	0.07
query5	0.48	0.48	0.48
query6	1.14	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.04	0.05
query9	0.57	0.50	0.51
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.13	0.12
query13	0.62	0.60	0.58
query14	0.79	0.78	0.79
query15	0.88	0.85	0.84
query16	0.36	0.36	0.35
query17	0.96	1.00	0.98
query18	0.26	0.25	0.25
query19	1.84	1.74	1.75
query20	0.01	0.02	0.01
query21	15.41	0.65	0.65
query22	3.95	7.26	1.48
query23	17.93	1.29	1.18
query24	1.51	0.20	0.20
query25	0.15	0.08	0.07
query26	0.27	0.17	0.17
query27	0.08	0.07	0.08
query28	13.81	0.96	0.95
query29	12.53	3.25	3.26
query30	0.26	0.05	0.06
query31	2.86	0.39	0.40
query32	3.26	0.47	0.48
query33	2.85	2.90	2.86
query34	15.49	4.34	4.31
query35	4.33	4.38	4.34
query36	0.67	0.47	0.48
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.15	0.15
query41	0.10	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.03	0.03
Total cold run time: 106.97 s
Total hot run time: 29.68 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 925ddb1af44c81a12857bcb74a81bf1d867399e2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

@englefly
Copy link
Contributor Author

englefly commented Apr 7, 2024

run p0

@englefly
Copy link
Contributor Author

englefly commented Apr 7, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c357be9679dafa05f76fe62ed7948829b309a12, data reload: false

------ Round 1 ----------------------------------
q1	17915	4233	4192	4192
q2	2336	193	183	183
q3	11112	1160	1350	1160
q4	11072	911	1033	911
q5	7799	3026	3009	3009
q6	223	143	137	137
q7	1132	638	603	603
q8	9423	2094	2040	2040
q9	6727	6205	6164	6164
q10	8408	3590	3517	3517
q11	415	241	238	238
q12	376	221	211	211
q13	17776	2901	2947	2901
q14	273	245	233	233
q15	526	482	468	468
q16	483	390	368	368
q17	963	906	930	906
q18	7197	6654	6376	6376
q19	1577	1558	1539	1539
q20	575	316	313	313
q21	3514	3161	3082	3082
q22	359	303	302	302
Total cold run time: 110181 ms
Total hot run time: 38853 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4056	4027	4043	4027
q2	331	220	219	219
q3	2980	2953	2989	2953
q4	1887	1849	1869	1849
q5	5232	5253	5230	5230
q6	207	125	126	125
q7	2203	1782	1795	1782
q8	3205	3300	3284	3284
q9	8451	8503	8488	8488
q10	3766	3837	3809	3809
q11	540	450	458	450
q12	718	561	534	534
q13	16739	2896	2920	2896
q14	292	252	259	252
q15	510	469	465	465
q16	451	386	409	386
q17	1725	1706	1685	1685
q18	7614	7217	7099	7099
q19	1651	1651	1645	1645
q20	1936	1734	1737	1734
q21	4994	4718	4751	4718
q22	498	428	426	426
Total cold run time: 69986 ms
Total hot run time: 54056 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180883 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c357be9679dafa05f76fe62ed7948829b309a12, data reload: false

query1	1220	1114	1118	1114
query2	6492	1762	1844	1762
query3	6663	215	219	215
query4	24333	21584	21446	21446
query5	4216	415	420	415
query6	279	182	174	174
query7	4602	301	297	297
query8	232	177	181	177
query9	8467	2227	2235	2227
query10	585	250	274	250
query11	14960	14720	14412	14412
query12	144	93	93	93
query13	1656	380	386	380
query14	8767	6831	6923	6831
query15	214	175	183	175
query16	7133	278	259	259
query17	1014	599	550	550
query18	1893	279	273	273
query19	189	156	153	153
query20	90	86	89	86
query21	196	129	132	129
query22	4973	4784	4718	4718
query23	33110	32564	32720	32564
query24	10938	3097	3076	3076
query25	666	370	387	370
query26	1896	159	162	159
query27	3058	322	326	322
query28	6751	1847	1815	1815
query29	1349	608	589	589
query30	317	166	163	163
query31	980	712	711	711
query32	96	59	58	58
query33	661	257	256	256
query34	948	487	502	487
query35	828	688	697	688
query36	970	838	825	825
query37	284	72	74	72
query38	3416	3411	3393	3393
query39	1560	1524	1543	1524
query40	293	133	143	133
query41	50	48	51	48
query42	115	107	105	105
query43	443	403	426	403
query44	1113	728	724	724
query45	275	253	252	252
query46	1065	788	765	765
query47	1844	1760	1770	1760
query48	378	301	310	301
query49	1151	371	381	371
query50	799	409	396	396
query51	6689	6640	6565	6565
query52	104	99	97	97
query53	367	291	289	289
query54	290	231	237	231
query55	86	74	75	74
query56	260	231	236	231
query57	1211	1117	1135	1117
query58	244	221	219	219
query59	2575	2558	2405	2405
query60	269	246	264	246
query61	112	114	110	110
query62	698	449	457	449
query63	310	286	282	282
query64	6394	3206	3212	3206
query65	3095	3019	2989	2989
query66	1512	337	326	326
query67	15626	15198	14904	14904
query68	7663	568	585	568
query69	530	321	314	314
query70	1316	1136	1080	1080
query71	499	290	276	276
query72	6405	2641	2404	2404
query73	804	327	326	326
query74	6703	6336	6347	6336
query75	3455	2274	2298	2274
query76	5440	1223	1234	1223
query77	696	249	251	249
query78	10807	10361	10065	10065
query79	8742	541	547	541
query80	1378	439	441	439
query81	504	243	243	243
query82	624	96	95	95
query83	216	173	171	171
query84	271	91	88	88
query85	1234	287	281	281
query86	370	311	297	297
query87	3713	3481	3474	3474
query88	3730	2361	2350	2350
query89	569	381	373	373
query90	2021	184	185	184
query91	140	105	104	104
query92	59	49	52	49
query93	6523	539	534	534
query94	1239	182	181	181
query95	437	327	321	321
query96	598	271	271	271
query97	2689	2541	2506	2506
query98	237	221	215	215
query99	1316	845	847	845
Total cold run time: 294442 ms
Total hot run time: 180883 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c357be9679dafa05f76fe62ed7948829b309a12, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.04	0.04
query4	1.70	0.06	0.07
query5	0.49	0.48	0.49
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.52	0.52
query10	0.59	0.58	0.56
query11	0.17	0.12	0.11
query12	0.14	0.12	0.13
query13	0.60	0.60	0.60
query14	0.77	0.80	0.76
query15	0.88	0.85	0.84
query16	0.35	0.36	0.36
query17	0.96	1.00	0.99
query18	0.24	0.24	0.28
query19	1.85	1.70	1.70
query20	0.01	0.01	0.01
query21	15.41	0.65	0.64
query22	3.90	7.56	1.91
query23	17.90	1.25	1.21
query24	1.49	0.20	0.19
query25	0.16	0.07	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.88	0.97	0.95
query29	12.57	3.27	3.23
query30	0.26	0.05	0.08
query31	2.86	0.40	0.39
query32	3.27	0.49	0.47
query33	2.84	2.90	2.89
query34	15.48	4.38	4.36
query35	4.39	4.40	4.40
query36	0.67	0.48	0.47
query37	0.17	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.04 s
Total hot run time: 30.24 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8c357be9679dafa05f76fe62ed7948829b309a12 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

@englefly
Copy link
Contributor Author

englefly commented Apr 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38543 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5e31f927786d50ada998f7ec2f744b5d6ef8781, data reload: false

------ Round 1 ----------------------------------
q1	17608	4578	4275	4275
q2	2021	188	194	188
q3	10522	1199	1183	1183
q4	10218	776	724	724
q5	7512	2747	2665	2665
q6	221	135	134	134
q7	1013	589	578	578
q8	9224	2065	2042	2042
q9	7940	6596	6561	6561
q10	8660	3538	3523	3523
q11	454	230	226	226
q12	510	213	205	205
q13	18804	2900	2945	2900
q14	270	243	231	231
q15	531	480	479	479
q16	510	379	375	375
q17	974	681	679	679
q18	7363	6838	6698	6698
q19	7778	1490	1523	1490
q20	700	321	314	314
q21	3484	2770	2787	2770
q22	355	309	303	303
Total cold run time: 116672 ms
Total hot run time: 38543 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4206	4255	4206
q2	367	261	264	261
q3	2974	2731	2765	2731
q4	1874	1649	1595	1595
q5	5352	5303	5313	5303
q6	209	125	123	123
q7	2240	1856	1894	1856
q8	3236	3370	3378	3370
q9	8609	8636	8794	8636
q10	4066	3909	3976	3909
q11	584	488	480	480
q12	785	631	618	618
q13	16405	3202	3140	3140
q14	301	294	276	276
q15	507	466	474	466
q16	501	441	442	441
q17	1814	1547	1538	1538
q18	8115	8010	7808	7808
q19	1736	1576	1584	1576
q20	1999	1860	1848	1848
q21	5122	5099	4882	4882
q22	558	475	475	475
Total cold run time: 71661 ms
Total hot run time: 55538 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181613 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5e31f927786d50ada998f7ec2f744b5d6ef8781, data reload: false

query1	1220	1131	345	345
query2	6998	2034	1885	1885
query3	6683	203	200	200
query4	36792	21548	21253	21253
query5	4163	392	396	392
query6	241	187	176	176
query7	4055	287	285	285
query8	223	176	187	176
query9	5753	2294	2299	2294
query10	381	233	231	231
query11	14775	14207	14237	14207
query12	135	86	85	85
query13	969	357	366	357
query14	9001	6866	7080	6866
query15	226	176	172	172
query16	7557	250	251	250
query17	1865	570	541	541
query18	1835	290	264	264
query19	186	145	148	145
query20	90	85	84	84
query21	202	123	126	123
query22	4991	4847	4845	4845
query23	33621	33035	33343	33035
query24	11360	3030	2996	2996
query25	498	365	423	365
query26	811	165	173	165
query27	2929	366	375	366
query28	7552	1951	1937	1937
query29	862	655	635	635
query30	321	180	178	178
query31	947	750	752	750
query32	61	56	59	56
query33	572	249	251	249
query34	1134	495	498	495
query35	874	726	737	726
query36	1054	928	927	927
query37	123	74	78	74
query38	3672	3707	3620	3620
query39	1679	1596	1585	1585
query40	188	135	136	135
query41	50	48	46	46
query42	114	101	104	101
query43	499	470	452	452
query44	1392	729	761	729
query45	296	293	261	261
query46	1096	752	702	702
query47	2018	1934	1919	1919
query48	367	293	300	293
query49	877	373	385	373
query50	787	387	414	387
query51	6863	6798	6825	6798
query52	102	92	93	92
query53	381	276	291	276
query54	277	245	230	230
query55	86	74	78	74
query56	253	230	240	230
query57	1282	1162	1188	1162
query58	230	207	209	207
query59	3023	2720	2601	2601
query60	254	235	239	235
query61	101	89	88	88
query62	582	445	431	431
query63	304	281	274	274
query64	4149	3704	3834	3704
query65	3096	3061	3035	3035
query66	710	330	314	314
query67	15571	15030	14834	14834
query68	10650	551	544	544
query69	615	306	299	299
query70	1400	1112	1178	1112
query71	499	264	273	264
query72	6627	2592	2413	2413
query73	1948	315	318	315
query74	7602	6389	6317	6317
query75	3596	2295	2276	2276
query76	6539	1144	1132	1132
query77	635	249	244	244
query78	10905	10155	10166	10155
query79	11521	517	516	516
query80	2451	410	423	410
query81	565	226	228	226
query82	252	93	89	89
query83	207	167	159	159
query84	261	79	83	79
query85	1095	271	258	258
query86	346	298	269	269
query87	3707	3573	3478	3478
query88	5875	2287	2298	2287
query89	496	374	367	367
query90	2383	185	186	185
query91	120	92	91	91
query92	59	45	46	45
query93	6680	496	501	496
query94	1565	182	176	176
query95	407	303	296	296
query96	625	268	263	263
query97	2643	2497	2505	2497
query98	232	218	214	214
query99	1123	868	823	823
Total cold run time: 314988 ms
Total hot run time: 181613 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5e31f927786d50ada998f7ec2f744b5d6ef8781, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.47	0.48
query6	1.12	0.65	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.56	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.11
query13	0.60	0.59	0.58
query14	0.77	0.76	0.78
query15	0.83	0.82	0.81
query16	0.37	0.35	0.35
query17	1.01	0.96	0.95
query18	0.22	0.24	0.25
query19	1.77	1.73	1.78
query20	0.01	0.01	0.01
query21	15.40	0.65	0.64
query22	4.48	6.20	2.24
query23	18.25	1.28	1.07
query24	1.62	0.24	0.24
query25	0.12	0.09	0.08
query26	0.27	0.16	0.16
query27	0.07	0.07	0.08
query28	13.40	0.99	0.98
query29	12.61	3.28	3.29
query30	0.25	0.06	0.06
query31	2.85	0.37	0.37
query32	3.31	0.46	0.45
query33	2.78	2.82	2.82
query34	17.22	4.41	4.41
query35	4.50	4.48	4.57
query36	0.64	0.47	0.46
query37	0.19	0.15	0.14
query38	0.15	0.14	0.15
query39	0.05	0.04	0.03
query40	0.17	0.15	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.03	0.03
Total cold run time: 109.21 s
Total hot run time: 30.41 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f5e31f927786d50ada998f7ec2f744b5d6ef8781 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

@englefly
Copy link
Contributor Author

englefly commented Apr 9, 2024

run buildall

@englefly englefly dismissed stale reviews from morrySnow and starocean999 via c6c35a8 April 9, 2024 07:04
@englefly
Copy link
Contributor Author

englefly commented Apr 9, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 9, 2024
@englefly
Copy link
Contributor Author

run buildall

morrySnow
morrySnow reviewed Apr 10, 2024
View reviewed changes
Copy link
Contributor

@morrySnow morrySnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a regression test to ensure we could process variant correctly after we refactor variant push down logic

fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/CommonSubExpressionOpt.java
project.getInputSlots(), project.getProjects());
project.setMultiLayerProjects(multiLayers);
project.child().accept(this, ctx);
if (!project.hasPushedDownToProjectionFunctions()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comments to explain why need this if branch

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 10, 2024
@englefly englefly merged commit 11bc9f2 into apache:master Apr 11, 2024
32 of 35 checks passed
dataroaring pushed a commit that referenced this pull request Apr 11, 2024
@englefly @dataroaring
[fix](nereids) fix a visitor bug in CommonSubExpressionOpt (#33154)
3ffe498 
* fix a bug in visitproject
* fix-variant
yiguolei pushed a commit that referenced this pull request Apr 12, 2024
@englefly @Doris-Extras
[fix](nereids) fix a visitor bug in CommonSubExpressionOpt (#33154)
22c4220 
* fix a bug in visitproject
* fix-variant
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

@LiBinfeng-01 LiBinfeng-01 Awaiting requested review from LiBinfeng-01

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@englefly @doris-robot @yiguolei @starocean999 @LiBinfeng-01 @morrySnow