Skip to content

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Aug 28, 2025

What problem does this PR solve?

when merge project, if the slot reference in parent project exists multiple times, and the slot reference's origin expression contains unique expression, then shouldn't merge these projects, otherwise be will calculate the unique function multiple time.

for example:

LogicalProject( k + k)
|
LogicalProject(random() as k)

after merge project, it will got:

LogicalProject(random() + random())

then be wil calcute twice random() for each row.

relate PR: #43546

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34323 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85857343e50cea147256b1fcb2caccdc5040fe83, data reload: false

------ Round 1 ----------------------------------
q1	17598	5197	5177	5177
q2	1989	321	215	215
q3	10243	1411	724	724
q4	10225	1054	539	539
q5	7535	2422	2374	2374
q6	189	168	134	134
q7	930	747	618	618
q8	9333	1328	1105	1105
q9	6993	5122	5083	5083
q10	6891	2411	1993	1993
q11	480	300	285	285
q12	354	363	219	219
q13	17761	3682	3052	3052
q14	252	251	226	226
q15	562	506	520	506
q16	445	435	378	378
q17	598	851	378	378
q18	7847	7116	7174	7116
q19	1113	958	575	575
q20	345	344	236	236
q21	3832	2592	2377	2377
q22	1084	1032	1013	1013
Total cold run time: 106599 ms
Total hot run time: 34323 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5154	5102	5127	5102
q2	250	333	232	232
q3	2136	2718	2298	2298
q4	1394	1805	1339	1339
q5	4199	4426	4645	4426
q6	228	208	136	136
q7	2026	2012	1804	1804
q8	2667	2592	2603	2592
q9	7531	7684	7162	7162
q10	3095	3327	3091	3091
q11	575	541	485	485
q12	677	791	669	669
q13	3453	3938	3307	3307
q14	287	311	278	278
q15	517	494	496	494
q16	453	528	463	463
q17	1354	1600	1353	1353
q18	7712	7729	7757	7729
q19	831	907	1107	907
q20	2062	2088	1899	1899
q21	5209	4499	4577	4499
q22	1155	1061	1029	1029
Total cold run time: 52965 ms
Total hot run time: 51294 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186825 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85857343e50cea147256b1fcb2caccdc5040fe83, data reload: false

query1	1063	444	416	416
query2	6564	1789	1720	1720
query3	6752	231	231	231
query4	26459	23766	23227	23227
query5	4479	649	552	552
query6	351	248	237	237
query7	4658	530	311	311
query8	306	269	255	255
query9	8655	2866	2844	2844
query10	492	356	298	298
query11	15301	15130	14782	14782
query12	183	124	125	124
query13	1671	562	439	439
query14	9395	5872	5724	5724
query15	223	201	181	181
query16	7653	668	459	459
query17	1233	746	623	623
query18	2048	434	333	333
query19	207	198	169	169
query20	193	125	122	122
query21	227	127	112	112
query22	4338	4425	4127	4127
query23	33699	32665	32975	32665
query24	8126	2365	2396	2365
query25	581	526	456	456
query26	1247	275	196	196
query27	2712	523	370	370
query28	4320	2259	2231	2231
query29	791	610	501	501
query30	292	231	202	202
query31	915	813	735	735
query32	93	90	81	81
query33	614	390	360	360
query34	806	921	524	524
query35	829	838	778	778
query36	986	1007	926	926
query37	142	120	94	94
query38	4076	4027	4036	4027
query39	1489	1421	1420	1420
query40	235	134	132	132
query41	73	71	63	63
query42	130	114	116	114
query43	490	514	496	496
query44	1340	868	848	848
query45	188	176	187	176
query46	878	1012	646	646
query47	1760	1793	1719	1719
query48	401	420	326	326
query49	749	524	429	429
query50	648	692	403	403
query51	4179	4174	4070	4070
query52	119	115	105	105
query53	248	282	200	200
query54	644	626	565	565
query55	100	94	100	94
query56	389	378	350	350
query57	1201	1205	1123	1123
query58	314	303	299	299
query59	2679	2744	2721	2721
query60	391	381	364	364
query61	236	164	160	160
query62	783	744	695	695
query63	232	192	193	192
query64	4466	1180	867	867
query65	4346	4179	4161	4161
query66	1139	445	349	349
query67	15461	15384	15120	15120
query68	8019	940	591	591
query69	498	355	300	300
query70	1275	1140	1142	1140
query71	486	347	330	330
query72	5830	5068	5283	5068
query73	782	676	369	369
query74	8921	8811	9243	8811
query75	3894	3093	2609	2609
query76	3733	1161	783	783
query77	811	418	340	340
query78	9493	9618	8866	8866
query79	2622	815	606	606
query80	680	581	544	544
query81	493	265	237	237
query82	461	151	114	114
query83	312	340	255	255
query84	304	114	99	99
query85	906	484	443	443
query86	353	314	285	285
query87	4343	4289	4225	4225
query88	3160	2237	2245	2237
query89	419	318	297	297
query90	1959	236	236	236
query91	163	172	140	140
query92	99	77	74	74
query93	1478	999	673	673
query94	716	421	335	335
query95	421	340	355	340
query96	489	586	281	281
query97	2682	2669	2623	2623
query98	254	243	233	233
query99	1451	1422	1337	1337
Total cold run time: 275495 ms
Total hot run time: 186825 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85857343e50cea147256b1fcb2caccdc5040fe83, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.08	0.09
query4	1.60	0.11	0.11
query5	0.45	0.42	0.42
query6	1.18	0.65	0.67
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.60	0.55	0.53
query10	0.59	0.58	0.58
query11	0.18	0.12	0.12
query12	0.16	0.13	0.13
query13	0.63	0.62	0.62
query14	0.79	0.87	0.84
query15	0.88	0.87	0.85
query16	0.39	0.42	0.39
query17	1.06	1.05	1.08
query18	0.22	0.20	0.21
query19	1.94	1.81	1.82
query20	0.01	0.02	0.02
query21	15.43	0.95	0.59
query22	0.79	1.21	0.74
query23	14.85	1.40	0.67
query24	6.73	1.58	1.42
query25	0.47	0.21	0.10
query26	0.53	0.15	0.14
query27	0.07	0.06	0.06
query28	10.26	0.91	0.42
query29	12.58	3.99	3.25
query30	3.15	3.11	3.00
query31	2.84	0.58	0.38
query32	3.24	0.56	0.48
query33	3.06	3.06	3.06
query34	16.04	5.48	4.81
query35	4.97	4.95	4.94
query36	0.71	0.51	0.50
query37	0.11	0.08	0.08
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.19	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.47 s
Total hot run time: 33.5 s

Comment on lines +46 to +47
if (ExpressionUtils.containsWindowExpression(parentProjects)
&& ExpressionUtils.containsWindowExpression(bottomProjects)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some comment to explain why cannot merge two project that contain window expression

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 42.11% (8/19) 🎉
Increment coverage report
Complete coverage report

@yujun777
Copy link
Contributor Author

run buildall

@yujun777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d972e8a44fe03ea47e32b9d37a6f2036c6762c8f, data reload: false

------ Round 1 ----------------------------------
q1	17600	5258	5062	5062
q2	2013	324	213	213
q3	10242	1280	732	732
q4	10232	1028	536	536
q5	7569	2479	2354	2354
q6	191	176	141	141
q7	941	784	631	631
q8	9385	1400	1125	1125
q9	6881	5115	5137	5115
q10	6878	2401	1997	1997
q11	506	315	288	288
q12	353	361	225	225
q13	17788	3638	3032	3032
q14	243	255	230	230
q15	571	506	507	506
q16	434	435	386	386
q17	630	867	360	360
q18	7686	7230	7115	7115
q19	1238	965	571	571
q20	356	359	249	249
q21	3917	3242	2469	2469
q22	1095	1041	989	989
Total cold run time: 106749 ms
Total hot run time: 34326 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5140	5124	5164	5124
q2	254	329	225	225
q3	2237	2721	2362	2362
q4	1353	1879	1387	1387
q5	4244	4492	4624	4492
q6	232	183	144	144
q7	2064	1984	1937	1937
q8	2648	2701	2676	2676
q9	7330	7414	7314	7314
q10	3157	3301	2888	2888
q11	582	509	509	509
q12	772	814	655	655
q13	3455	3954	3229	3229
q14	315	319	275	275
q15	528	479	492	479
q16	442	493	445	445
q17	1223	1574	1433	1433
q18	7799	7600	7660	7600
q19	833	814	943	814
q20	1981	1918	1832	1832
q21	4927	4354	4425	4354
q22	1102	1057	1004	1004
Total cold run time: 52618 ms
Total hot run time: 51178 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187179 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d972e8a44fe03ea47e32b9d37a6f2036c6762c8f, data reload: false

query1	1078	457	418	418
query2	6605	1776	1814	1776
query3	6756	235	230	230
query4	26634	23583	23410	23410
query5	4391	637	511	511
query6	341	259	223	223
query7	4654	525	308	308
query8	310	263	264	263
query9	8622	2934	2956	2934
query10	532	377	293	293
query11	15794	15331	14725	14725
query12	194	115	117	115
query13	1668	566	449	449
query14	9672	5884	5828	5828
query15	214	198	182	182
query16	7662	675	483	483
query17	1244	756	613	613
query18	2059	442	342	342
query19	208	200	171	171
query20	146	132	120	120
query21	221	128	111	111
query22	4187	4231	4135	4135
query23	34003	33088	32968	32968
query24	8164	2404	2361	2361
query25	567	523	471	471
query26	1235	281	168	168
query27	2754	506	354	354
query28	4363	2296	2267	2267
query29	758	597	497	497
query30	281	225	198	198
query31	922	807	728	728
query32	99	81	78	78
query33	587	400	371	371
query34	820	854	532	532
query35	846	828	740	740
query36	994	1026	909	909
query37	128	122	102	102
query38	3997	4053	4021	4021
query39	1492	1424	1428	1424
query40	234	144	138	138
query41	72	70	68	68
query42	137	123	120	120
query43	527	531	479	479
query44	1360	906	867	867
query45	185	176	175	175
query46	879	1027	653	653
query47	1794	1826	1781	1781
query48	387	440	325	325
query49	742	513	419	419
query50	655	692	400	400
query51	4149	4175	4262	4175
query52	119	115	109	109
query53	250	272	212	212
query54	626	628	553	553
query55	96	95	95	95
query56	356	339	340	339
query57	1202	1205	1160	1160
query58	294	332	278	278
query59	2630	2733	2586	2586
query60	365	361	347	347
query61	167	157	162	157
query62	816	729	673	673
query63	228	194	196	194
query64	4378	1154	839	839
query65	4304	4248	4249	4248
query66	1090	445	357	357
query67	15562	15558	15241	15241
query68	6858	944	598	598
query69	496	353	310	310
query70	1237	1112	1116	1112
query71	457	359	334	334
query72	6004	4965	5142	4965
query73	674	602	361	361
query74	8897	9125	8623	8623
query75	3180	3073	2643	2643
query76	3265	1147	740	740
query77	498	454	355	355
query78	9539	9978	8886	8886
query79	2445	814	601	601
query80	639	574	516	516
query81	504	270	232	232
query82	217	141	114	114
query83	265	265	240	240
query84	265	113	98	98
query85	861	461	443	443
query86	380	327	275	275
query87	4358	4353	4256	4256
query88	3740	2244	2231	2231
query89	388	338	301	301
query90	1991	240	231	231
query91	177	163	133	133
query92	92	85	74	74
query93	1886	1015	670	670
query94	705	427	338	338
query95	421	349	338	338
query96	501	581	281	281
query97	2603	2679	2578	2578
query98	241	215	211	211
query99	1353	1381	1274	1274
Total cold run time: 273943 ms
Total hot run time: 187179 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d972e8a44fe03ea47e32b9d37a6f2036c6762c8f, data reload: false

query1	0.06	0.05	0.04
query2	0.09	0.06	0.06
query3	0.25	0.09	0.08
query4	1.60	0.12	0.11
query5	0.44	0.42	0.43
query6	1.15	0.65	0.66
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.60	0.54	0.53
query10	0.58	0.59	0.58
query11	0.18	0.11	0.11
query12	0.16	0.12	0.13
query13	0.63	0.63	0.62
query14	0.81	0.84	0.85
query15	0.90	0.87	0.88
query16	0.39	0.40	0.38
query17	1.09	1.08	1.07
query18	0.24	0.20	0.21
query19	1.96	1.88	1.85
query20	0.01	0.02	0.01
query21	15.40	0.98	0.58
query22	0.80	1.30	0.78
query23	14.78	1.40	0.63
query24	6.64	1.15	1.73
query25	0.49	0.27	0.10
query26	0.58	0.16	0.15
query27	0.06	0.06	0.05
query28	10.28	0.92	0.44
query29	12.59	3.86	3.22
query30	3.13	3.00	3.02
query31	2.82	0.58	0.38
query32	3.26	0.56	0.48
query33	3.14	3.09	3.14
query34	16.11	5.53	4.83
query35	4.92	4.97	4.93
query36	0.70	0.51	0.51
query37	0.10	0.08	0.07
query38	0.06	0.04	0.05
query39	0.03	0.04	0.03
query40	0.19	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 107.49 s
Total hot run time: 33.32 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 94.74% (18/19) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit 97ac757 into apache:master Aug 29, 2025
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants