Skip to content

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jul 16, 2025

What problem does this PR solve?

Related PR: #40680

In #40680, some logic was implemented in the old optimizer, this PR reimplements part of the logic in the nereids.

Problem Summary:
support temporary table in nereids

Release note

support temporary table in nereids

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jul 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Jul 16, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

------ Round 1 ----------------------------------
q1	17583	5286	5058	5058
q2	1960	275	180	180
q3	10314	1294	719	719
q4	10256	1002	512	512
q5	8318	2368	2383	2368
q6	186	161	129	129
q7	902	757	618	618
q8	9318	1359	1124	1124
q9	6855	5103	5077	5077
q10	6881	2355	1949	1949
q11	484	299	276	276
q12	342	351	212	212
q13	17771	3628	3124	3124
q14	218	231	212	212
q15	546	482	482	482
q16	426	441	372	372
q17	588	896	359	359
q18	7536	7164	7108	7108
q19	1583	958	568	568
q20	335	338	223	223
q21	3640	3150	2333	2333
q22	1045	1005	948	948
Total cold run time: 107087 ms
Total hot run time: 33951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5245	5150	5155	5150
q2	268	320	212	212
q3	2176	2640	2298	2298
q4	1368	1770	1350	1350
q5	4220	4512	4564	4512
q6	232	174	126	126
q7	1985	1963	1837	1837
q8	2721	2488	2528	2488
q9	7235	7281	7299	7281
q10	3108	3379	2856	2856
q11	567	517	504	504
q12	697	842	846	842
q13	3539	4010	3264	3264
q14	333	314	289	289
q15	516	484	475	475
q16	460	497	438	438
q17	1199	1558	1378	1378
q18	8024	7717	7789	7717
q19	777	808	888	808
q20	2132	2046	1907	1907
q21	4764	4388	4312	4312
q22	1062	1014	969	969
Total cold run time: 52628 ms
Total hot run time: 51013 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186579 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

query1	984	382	406	382
query2	6505	1688	1599	1599
query3	6745	210	210	210
query4	26412	23341	23480	23341
query5	4325	571	424	424
query6	297	211	221	211
query7	4626	493	285	285
query8	262	225	215	215
query9	8641	2649	2647	2647
query10	482	348	286	286
query11	15726	15057	14774	14774
query12	166	103	101	101
query13	1655	521	406	406
query14	9624	5825	5755	5755
query15	210	192	190	190
query16	7647	619	440	440
query17	1183	714	564	564
query18	2017	395	292	292
query19	191	183	155	155
query20	119	113	113	113
query21	211	121	101	101
query22	4113	4408	4136	4136
query23	34046	33209	33087	33087
query24	8473	2391	2354	2354
query25	568	472	380	380
query26	1240	262	143	143
query27	2762	503	338	338
query28	4327	2152	2115	2115
query29	713	546	461	461
query30	281	219	185	185
query31	952	848	743	743
query32	73	67	65	65
query33	551	381	302	302
query34	793	828	514	514
query35	797	809	734	734
query36	989	986	930	930
query37	104	91	71	71
query38	4197	4233	4055	4055
query39	1534	1433	1416	1416
query40	210	115	103	103
query41	60	55	52	52
query42	117	104	104	104
query43	503	513	491	491
query44	1325	835	837	835
query45	179	171	168	168
query46	836	1016	621	621
query47	1816	1832	1764	1764
query48	389	429	323	323
query49	748	490	411	411
query50	642	690	434	434
query51	5509	5501	5406	5406
query52	104	105	98	98
query53	223	249	181	181
query54	554	582	514	514
query55	80	82	84	82
query56	320	290	308	290
query57	1217	1193	1124	1124
query58	259	249	254	249
query59	2580	2678	2419	2419
query60	319	315	290	290
query61	126	121	123	121
query62	832	710	671	671
query63	215	188	190	188
query64	4277	1066	661	661
query65	4244	4169	4162	4162
query66	1057	411	327	327
query67	16044	15751	15440	15440
query68	7884	890	532	532
query69	473	306	264	264
query70	1225	1155	1150	1150
query71	460	313	298	298
query72	5619	4770	4828	4770
query73	671	621	346	346
query74	8965	8838	8877	8838
query75	3761	3184	2694	2694
query76	3601	1177	723	723
query77	794	378	298	298
query78	10052	10193	9390	9390
query79	1969	811	580	580
query80	601	522	443	443
query81	484	251	222	222
query82	414	128	97	97
query83	251	253	230	230
query84	247	108	99	99
query85	801	357	312	312
query86	337	313	290	290
query87	4507	4561	4346	4346
query88	3261	2319	2294	2294
query89	393	323	297	297
query90	1940	209	205	205
query91	134	141	122	122
query92	77	61	98	61
query93	1048	928	589	589
query94	671	390	314	314
query95	378	290	284	284
query96	492	569	277	277
query97	2762	2733	2696	2696
query98	227	202	202	202
query99	1656	1429	1267	1267
Total cold run time: 275481 ms
Total hot run time: 186579 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.62	0.11	0.11
query5	0.44	0.43	0.44
query6	1.18	0.66	0.66
query7	0.03	0.01	0.02
query8	0.05	0.04	0.04
query9	0.60	0.51	0.51
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.16	0.11	0.11
query13	0.63	0.61	0.62
query14	0.80	0.81	0.82
query15	0.90	0.88	0.86
query16	0.38	0.39	0.38
query17	1.04	1.03	1.07
query18	0.23	0.21	0.21
query19	1.96	1.79	1.83
query20	0.01	0.02	0.01
query21	15.40	0.92	0.53
query22	0.74	1.16	0.94
query23	14.73	1.36	0.60
query24	7.09	1.32	0.33
query25	0.35	0.14	0.18
query26	0.66	0.17	0.15
query27	0.05	0.05	0.05
query28	9.24	0.89	0.45
query29	12.55	3.97	3.37
query30	3.05	2.97	2.95
query31	2.83	0.58	0.38
query32	3.22	0.56	0.47
query33	3.04	3.08	3.10
query34	16.13	5.40	4.83
query35	4.84	4.87	4.83
query36	0.69	0.51	0.49
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.25 s
Total hot run time: 32.33 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

@zddr
Copy link
Contributor Author

zddr commented Jul 16, 2025

run feut

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

@zddr
Copy link
Contributor Author

zddr commented Jul 18, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34602 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

------ Round 1 ----------------------------------
q1	17589	5240	5141	5141
q2	1960	301	196	196
q3	10415	1336	720	720
q4	10277	1021	518	518
q5	8087	2409	2438	2409
q6	197	175	141	141
q7	941	781	631	631
q8	9361	1336	1123	1123
q9	6848	5096	5086	5086
q10	6912	2395	1981	1981
q11	506	318	294	294
q12	361	402	239	239
q13	17790	3767	3212	3212
q14	252	261	240	240
q15	591	525	537	525
q16	440	450	406	406
q17	618	882	387	387
q18	7669	7162	7249	7162
q19	1250	975	604	604
q20	363	359	236	236
q21	4265	3207	2365	2365
q22	1055	1051	986	986
Total cold run time: 107747 ms
Total hot run time: 34602 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5344	5211	5135	5135
q2	268	352	246	246
q3	2168	2692	2307	2307
q4	1384	1797	1331	1331
q5	4493	4488	4435	4435
q6	228	173	140	140
q7	2025	2013	1892	1892
q8	2677	2579	2554	2554
q9	7318	7306	7276	7276
q10	3182	3269	2930	2930
q11	593	525	507	507
q12	694	769	625	625
q13	3571	3977	3574	3574
q14	280	310	300	300
q15	561	514	505	505
q16	433	503	493	493
q17	1251	1590	1398	1398
q18	8043	7539	7480	7480
q19	845	869	1058	869
q20	2130	2120	1930	1930
q21	5116	4667	4600	4600
q22	1101	1030	1004	1004
Total cold run time: 53705 ms
Total hot run time: 51531 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

query1	1010	419	424	419
query2	6557	1836	1794	1794
query3	6752	224	229	224
query4	26631	23795	22994	22994
query5	4447	721	552	552
query6	320	243	227	227
query7	4657	512	309	309
query8	397	346	332	332
query9	8683	2676	2694	2676
query10	521	378	308	308
query11	15915	15400	14906	14906
query12	187	135	135	135
query13	1686	585	454	454
query14	10016	6178	6087	6087
query15	228	215	184	184
query16	7646	640	498	498
query17	1199	717	594	594
query18	2047	434	337	337
query19	210	223	187	187
query20	159	130	139	130
query21	239	143	126	126
query22	4023	4342	4105	4105
query23	34207	33130	33204	33130
query24	8536	2461	2460	2460
query25	547	490	459	459
query26	1240	285	164	164
query27	2736	519	363	363
query28	4422	2220	2173	2173
query29	752	601	471	471
query30	315	242	221	221
query31	998	924	815	815
query32	83	109	77	77
query33	606	423	351	351
query34	858	851	540	540
query35	813	845	763	763
query36	1013	996	940	940
query37	122	111	87	87
query38	4129	4165	4174	4165
query39	1557	1487	1467	1467
query40	240	142	129	129
query41	109	103	133	103
query42	138	117	118	117
query43	545	537	499	499
query44	1375	860	861	860
query45	203	193	179	179
query46	898	1041	673	673
query47	1803	1838	1766	1766
query48	399	450	335	335
query49	852	600	490	490
query50	699	716	447	447
query51	5530	5599	5468	5468
query52	121	123	113	113
query53	257	289	209	209
query54	614	611	527	527
query55	90	88	92	88
query56	349	373	345	345
query57	1220	1225	1187	1187
query58	327	308	311	308
query59	2591	2763	2606	2606
query60	371	370	358	358
query61	154	151	148	148
query62	818	748	699	699
query63	252	220	212	212
query64	4351	1070	748	748
query65	4310	4308	4205	4205
query66	1203	603	562	562
query67	15976	15797	15547	15547
query68	8572	916	551	551
query69	502	326	305	305
query70	1253	1186	1099	1099
query71	514	356	330	330
query72	5562	4792	4876	4792
query73	766	665	371	371
query74	9340	8943	8782	8782
query75	4084	3243	2892	2892
query76	3746	1191	775	775
query77	853	460	369	369
query78	10134	10252	9406	9406
query79	2766	807	590	590
query80	818	576	510	510
query81	484	269	254	254
query82	418	137	110	110
query83	320	286	274	274
query84	301	119	92	92
query85	812	408	355	355
query86	351	314	309	309
query87	4580	4527	4358	4358
query88	2999	2406	2367	2367
query89	444	346	314	314
query90	2098	239	236	236
query91	156	151	126	126
query92	95	73	71	71
query93	2479	972	616	616
query94	692	433	324	324
query95	402	319	349	319
query96	503	581	291	291
query97	2714	2803	2688	2688
query98	243	221	218	218
query99	1477	1439	1303	1303
Total cold run time: 282737 ms
Total hot run time: 190549 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edf19a61bd10dd5aa280498b98c3325082bbba9e, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.46	0.42	0.46
query6	1.17	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.53	0.52
query10	0.59	0.57	0.58
query11	0.17	0.12	0.12
query12	0.16	0.13	0.12
query13	0.63	0.63	0.61
query14	0.80	0.82	0.81
query15	0.92	0.88	0.89
query16	0.39	0.39	0.41
query17	1.07	1.06	1.08
query18	0.23	0.21	0.22
query19	1.94	1.86	1.85
query20	0.02	0.02	0.02
query21	15.36	0.89	0.54
query22	0.77	1.21	0.72
query23	14.83	1.36	0.64
query24	7.69	0.81	1.11
query25	0.47	0.32	0.07
query26	0.59	0.17	0.15
query27	0.07	0.06	0.05
query28	9.32	0.94	0.47
query29	12.52	3.95	3.32
query30	3.15	3.10	3.00
query31	2.84	0.61	0.38
query32	3.23	0.56	0.50
query33	3.13	3.08	3.08
query34	16.01	5.45	4.77
query35	4.82	4.85	4.84
query36	0.69	0.50	0.50
query37	0.10	0.08	0.08
query38	0.06	0.05	0.05
query39	0.05	0.03	0.03
query40	0.20	0.16	0.15
query41	0.09	0.04	0.04
query42	0.05	0.04	0.03
query43	0.04	0.04	0.05
Total cold run time: 107.27 s
Total hot run time: 32.83 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit 368f530 into apache:master Jul 23, 2025
33 checks passed
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2025
Related PR: apache#40680 

In apache#40680, some logic was
implemented in the old optimizer, this PR reimplements part of the logic
in the nereids.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants