Skip to content

[refactor](fe) Refine volatile expression handling#63403

Merged
starocean999 merged 3 commits into
apache:masterfrom
yujun777:opt-volatile
May 20, 2026
Merged

[refactor](fe) Refine volatile expression handling#63403
starocean999 merged 3 commits into
apache:masterfrom
yujun777:opt-volatile

Conversation

@yujun777
Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #62698

Problem Summary:

PR #62698 introduced the UDF volatility property and added VolatileExpression / VolatileIdentity so volatile UDF calls can carry per-call identity and avoid unsafe optimizer rewrites.

This PR is a follow-up refactoring for that change. It removes duplicated unique identity state from UniqueFunction, keeps VolatileIdentity as the single identity holder, moves volatile-expression helper methods into ExpressionTrait, and expands the former AddProjectForUniqueFunction rewrite to operate on volatile expressions rather than only unique functions.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • AddProjectForVolatileExpressionTest
      • UdfVolatilityTest
      • UniqueFunctionTest
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Move volatile checks onto ExpressionTrait and let volatile expressions share the same caller path. Clean up UniqueFunction identity storage to rely on VolatileIdentity and expand the project-alias rewrite to volatile expressions.

Key changes:

- Use VolatileIdentity as the single identity holder for UniqueFunction and subclasses.

- Rename AddProjectForUniqueFunction to AddProjectForVolatileExpression and apply it to volatile expressions.

- Move containsVolatileExpression/isVolatile helpers to ExpressionTrait and update related call sites.

Unit Test:

- AddProjectForVolatileExpressionTest

- UdfVolatilityTest

- UniqueFunctionTest
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777
Copy link
Copy Markdown
Contributor Author

/review

Fix the FE checkstyle failure caused by the import order in ExpressionTrait.

Key changes:

- Reorder ExpressionTrait imports according to checkstyle.

Unit Test:

- ./build.sh --fe
@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

Clarify that UniqueFunction overrides the volatile expression unique-id handling API with a covariant return type.

Key changes:

- Add @OverRide to UniqueFunction.withIgnoreUniqueId.

Unit Test:

- ./build.sh --fe
@yujun777
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 31278 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b6135b9506b76c3bbe4da62571a2d689a46ad71, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17796	3919	3871	3871
q2	q3	10789	1396	791	791
q4	4684	476	342	342
q5	7583	2250	2071	2071
q6	285	181	148	148
q7	920	788	646	646
q8	9354	1755	1588	1588
q9	6780	4903	4888	4888
q10	6427	2135	1813	1813
q11	435	271	240	240
q12	692	420	303	303
q13	18183	3414	2738	2738
q14	266	254	247	247
q15	q16	824	774	704	704
q17	1032	847	890	847
q18	6964	5766	5591	5591
q19	1213	1220	1235	1220
q20	545	436	293	293
q21	6074	2626	2672	2626
q22	449	367	311	311
Total cold run time: 101295 ms
Total hot run time: 31278 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4678	4578	4688	4578
q2	q3	4848	5231	4678	4678
q4	2173	2177	1413	1413
q5	4814	4792	4587	4587
q6	233	177	133	133
q7	1894	1733	1504	1504
q8	2380	1909	1879	1879
q9	7129	7162	7181	7162
q10	4474	4427	4004	4004
q11	530	387	348	348
q12	711	719	506	506
q13	3035	3470	2789	2789
q14	267	278	252	252
q15	q16	678	696	617	617
q17	1267	1234	1234	1234
q18	7341	6796	6737	6737
q19	1158	1109	1126	1109
q20	2215	2218	1929	1929
q21	5282	4594	4513	4513
q22	523	463	398	398
Total cold run time: 55630 ms
Total hot run time: 50370 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169490 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b6135b9506b76c3bbe4da62571a2d689a46ad71, data reload: false

query5	4319	656	519	519
query6	328	211	203	203
query7	4232	556	294	294
query8	332	226	216	216
query9	8813	3993	3979	3979
query10	454	340	288	288
query11	5781	2398	2237	2237
query12	183	146	130	130
query13	1277	621	430	430
query14	6023	5440	5062	5062
query14_1	4360	4374	4341	4341
query15	214	207	182	182
query16	994	466	465	465
query17	1161	752	619	619
query18	2550	493	364	364
query19	218	210	171	171
query20	139	133	133	133
query21	216	142	128	128
query22	13662	13557	13426	13426
query23	17121	16219	16079	16079
query23_1	16230	16195	16208	16195
query24	7466	1765	1304	1304
query24_1	1321	1311	1310	1310
query25	573	517	452	452
query26	1311	326	171	171
query27	2691	585	335	335
query28	4431	1961	1940	1940
query29	1039	636	523	523
query30	309	247	208	208
query31	1139	1098	942	942
query32	91	78	76	76
query33	540	374	314	314
query34	1179	1121	648	648
query35	770	807	669	669
query36	1318	1354	1212	1212
query37	154	102	89	89
query38	3236	3135	3067	3067
query39	940	943	901	901
query39_1	873	876	893	876
query40	230	141	128	128
query41	66	64	63	63
query42	111	117	111	111
query43	320	324	286	286
query44	
query45	211	203	195	195
query46	1044	1178	729	729
query47	2293	2285	2177	2177
query48	419	388	289	289
query49	625	484	386	386
query50	973	348	252	252
query51	4310	4267	4328	4267
query52	105	105	92	92
query53	255	282	211	211
query54	315	274	249	249
query55	92	93	83	83
query56	316	318	304	304
query57	1497	1438	1347	1347
query58	318	279	274	274
query59	1565	1602	1446	1446
query60	328	337	315	315
query61	162	161	152	152
query62	675	628	571	571
query63	257	204	210	204
query64	2422	786	615	615
query65	
query66	1724	469	353	353
query67	29962	29937	29868	29868
query68	
query69	466	338	308	308
query70	1045	1023	990	990
query71	294	285	269	269
query72	2983	2698	2377	2377
query73	861	749	436	436
query74	5100	4920	4742	4742
query75	2695	2605	2256	2256
query76	2289	1161	802	802
query77	400	402	336	336
query78	12169	12205	11572	11572
query79	1461	972	733	733
query80	656	531	446	446
query81	451	274	237	237
query82	1311	157	130	130
query83	361	277	247	247
query84	257	133	115	115
query85	887	566	451	451
query86	397	347	311	311
query87	3413	3382	3228	3228
query88	3508	2645	2629	2629
query89	456	384	337	337
query90	1981	183	175	175
query91	175	172	134	134
query92	79	74	70	70
query93	1553	1481	854	854
query94	528	357	307	307
query95	665	473	344	344
query96	1063	785	348	348
query97	2695	2703	2567	2567
query98	238	228	233	228
query99	1087	1126	988	988
Total cold run time: 253089 ms
Total hot run time: 169490 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 31116 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b6135b9506b76c3bbe4da62571a2d689a46ad71, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17705	3859	3839	3839
q2	q3	10784	1369	791	791
q4	4688	461	337	337
q5	7576	2230	2086	2086
q6	342	176	144	144
q7	920	783	619	619
q8	9372	1580	1573	1573
q9	6888	4881	4898	4881
q10	6464	2085	1790	1790
q11	437	274	239	239
q12	694	444	290	290
q13	18156	3400	2781	2781
q14	264	254	235	235
q15	q16	812	786	700	700
q17	950	938	882	882
q18	6766	5676	5567	5567
q19	1211	1314	1160	1160
q20	563	419	266	266
q21	6095	2684	2619	2619
q22	444	391	317	317
Total cold run time: 101131 ms
Total hot run time: 31116 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4639	4503	4623	4503
q2	q3	4862	5188	4630	4630
q4	2123	2170	1384	1384
q5	4813	4673	4633	4633
q6	244	178	129	129
q7	1851	1740	1558	1558
q8	2344	1883	1866	1866
q9	7166	7130	7122	7122
q10	4464	4387	4006	4006
q11	528	378	351	351
q12	699	728	503	503
q13	3024	3405	2795	2795
q14	262	271	249	249
q15	q16	668	689	607	607
q17	1249	1219	1218	1218
q18	7234	6689	6918	6689
q19	1144	1120	1094	1094
q20	2206	2217	1917	1917
q21	5281	4548	4447	4447
q22	502	472	416	416
Total cold run time: 55303 ms
Total hot run time: 50117 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169291 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b6135b9506b76c3bbe4da62571a2d689a46ad71, data reload: false

query5	4316	639	533	533
query6	338	217	195	195
query7	4219	545	297	297
query8	326	233	224	224
query9	8787	3991	3970	3970
query10	466	327	293	293
query11	5806	2390	2223	2223
query12	175	132	130	130
query13	1276	629	461	461
query14	6070	5314	5054	5054
query14_1	4311	4339	4306	4306
query15	212	203	181	181
query16	1005	434	427	427
query17	1130	713	567	567
query18	2549	480	357	357
query19	216	198	160	160
query20	135	128	129	128
query21	213	138	116	116
query22	13554	13468	13303	13303
query23	17190	16402	16050	16050
query23_1	16119	16166	16190	16166
query24	7525	1753	1283	1283
query24_1	1304	1299	1296	1296
query25	574	478	443	443
query26	1336	311	176	176
query27	2713	543	338	338
query28	4440	1962	1941	1941
query29	987	623	485	485
query30	305	240	196	196
query31	1112	1063	948	948
query32	91	71	74	71
query33	537	356	293	293
query34	1146	1108	626	626
query35	755	768	679	679
query36	1350	1332	1214	1214
query37	151	107	90	90
query38	3207	3150	3033	3033
query39	932	916	895	895
query39_1	872	913	887	887
query40	228	143	127	127
query41	65	70	63	63
query42	111	112	106	106
query43	319	326	282	282
query44	
query45	209	200	194	194
query46	1025	1144	737	737
query47	2271	2353	2265	2265
query48	420	410	295	295
query49	631	486	384	384
query50	983	356	246	246
query51	4376	4309	4241	4241
query52	106	109	99	99
query53	261	283	210	210
query54	329	281	255	255
query55	92	91	84	84
query56	309	323	319	319
query57	1466	1457	1357	1357
query58	303	288	273	273
query59	1508	1587	1411	1411
query60	369	331	294	294
query61	152	156	149	149
query62	676	631	557	557
query63	248	199	207	199
query64	2424	810	622	622
query65	
query66	1714	476	350	350
query67	29872	29869	29746	29746
query68	
query69	459	335	302	302
query70	1008	942	995	942
query71	312	270	270	270
query72	2978	2756	2599	2599
query73	840	754	446	446
query74	5060	4863	4714	4714
query75	2692	2593	2287	2287
query76	2297	1153	762	762
query77	396	423	332	332
query78	12082	12160	11481	11481
query79	1467	975	779	779
query80	710	564	482	482
query81	464	281	252	252
query82	1222	159	122	122
query83	325	288	256	256
query84	264	144	115	115
query85	962	606	554	554
query86	391	349	307	307
query87	3403	3367	3197	3197
query88	3518	2652	2640	2640
query89	452	386	335	335
query90	1870	184	182	182
query91	175	164	138	138
query92	78	79	74	74
query93	1459	1515	934	934
query94	553	363	290	290
query95	658	385	437	385
query96	1000	773	353	353
query97	2679	2691	2563	2563
query98	234	224	242	224
query99	1125	1088	977	977
Total cold run time: 252347 ms
Total hot run time: 169291 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 88.89% (32/36) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 20, 2026
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@starocean999
Copy link
Copy Markdown
Contributor

/review

@starocean999 starocean999 merged commit 820f51b into apache:master May 20, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.x dev/4.1.x-conflict reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants