[opt](explode) Optimize explode_outer and posexplode by jacktengg · Pull Request #62069 · apache/doris

jacktengg · 2026-04-03T01:48:12Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Improvement of #60352, optimize explode_outer, posexplode and posexplode_outer.

The approach is a single-pass algorithm: walk through child rows, accumulating contiguous segments into the output, then when hitting a null/empty row or reaching the end, flush the segment using bulk operations. For outer-null rows, insert a NULL and copy the non-table-function columns directly. This naturally handles both outer and
non-outer modes since non-outer mode just won't produce any null outputs. For posexplode, generate position indices alongside this.

Performance test result.
Create and populate a large test table. Use 100,000,000 base rows, each with an array of 10 INT elements, producing 1,000,000,000 total exploded output rows.

Function	slow path	optimized	SpeedUp
explode	9734 ms	5105 ms	90%
explode_outer	9685 ms	5064 ms	91%
posexplode	11021 ms	5963 ms	84%
posexplode_outer	14088 ms	5903 ms	138%

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-04-03T01:48:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

jacktengg · 2026-04-03T01:48:21Z

run buildall

doris-robot · 2026-04-03T02:17:26Z

TPC-H: Total hot run time: 29612 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b0a11863c538641d4fb8659018a9c71c7e60b83, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17789	3766	3789	3766
q2	q3	10680	863	605	605
q4	4672	472	371	371
q5	7442	1346	1171	1171
q6	189	164	137	137
q7	924	974	768	768
q8	9301	1455	1283	1283
q9	5652	5364	5331	5331
q10	6301	2055	1796	1796
q11	495	281	287	281
q12	898	699	522	522
q13	18021	2817	2183	2183
q14	289	284	266	266
q15	q16	910	865	806	806
q17	1095	1223	761	761
q18	6469	5706	5572	5572
q19	1352	1283	1110	1110
q20	618	549	411	411
q21	4720	2476	2098	2098
q22	527	441	374	374
Total cold run time: 98344 ms
Total hot run time: 29612 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4530	4380	4318	4318
q2	q3	4652	4773	4220	4220
q4	2082	2082	1375	1375
q5	4900	5007	5214	5007
q6	211	177	142	142
q7	2039	1785	1629	1629
q8	3300	3122	3085	3085
q9	8290	8431	8226	8226
q10	4474	4460	4276	4276
q11	581	421	400	400
q12	715	730	497	497
q13	2828	3109	2357	2357
q14	290	308	274	274
q15	q16	777	791	697	697
q17	1312	1238	1275	1238
q18	8104	7059	6991	6991
q19	1138	1086	1173	1086
q20	2414	2426	2006	2006
q21	6133	5348	4800	4800
q22	574	496	496	496
Total cold run time: 59344 ms
Total hot run time: 53120 ms

doris-robot · 2026-04-03T02:29:01Z

TPC-DS: Total hot run time: 180152 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b0a11863c538641d4fb8659018a9c71c7e60b83, data reload: false

query5	4327	655	522	522
query6	343	230	212	212
query7	4276	575	331	331
query8	339	240	218	218
query9	8737	3971	3921	3921
query10	526	396	342	342
query11	6769	5471	5126	5126
query12	189	132	135	132
query13	1369	626	461	461
query14	5711	5180	4769	4769
query14_1	4197	4198	4242	4198
query15	215	208	188	188
query16	1002	472	465	465
query17	1174	781	643	643
query18	2445	497	373	373
query19	246	217	185	185
query20	139	128	134	128
query21	220	146	122	122
query22	14888	14538	14920	14538
query23	18295	17134	16706	16706
query23_1	16937	16734	16664	16664
query24	7465	1770	1360	1360
query24_1	1360	1366	1369	1366
query25	575	485	429	429
query26	1238	328	186	186
query27	2658	626	358	358
query28	4451	1943	1920	1920
query29	981	660	555	555
query30	304	240	187	187
query31	1086	1060	955	955
query32	92	71	67	67
query33	525	363	286	286
query34	1198	1127	684	684
query35	740	765	661	661
query36	1303	1250	1060	1060
query37	151	92	84	84
query38	3120	3067	2989	2989
query39	914	895	862	862
query39_1	849	842	830	830
query40	241	159	138	138
query41	62	58	57	57
query42	279	269	272	269
query43	320	318	279	279
query44	
query45	210	200	192	192
query46	1165	1258	803	803
query47	2347	2375	2213	2213
query48	414	429	299	299
query49	653	535	430	430
query50	722	288	216	216
query51	4348	4305	4226	4226
query52	285	289	266	266
query53	326	351	275	275
query54	323	292	282	282
query55	103	98	89	89
query56	326	332	349	332
query57	1703	1653	1480	1480
query58	315	282	283	282
query59	2893	3024	2719	2719
query60	332	335	335	335
query61	159	153	160	153
query62	694	618	575	575
query63	314	273	270	270
query64	5362	1459	1079	1079
query65	
query66	1447	485	388	388
query67	24326	24252	24172	24172
query68	
query69	455	360	324	324
query70	1022	993	1023	993
query71	360	329	317	317
query72	3100	2878	2630	2630
query73	848	784	474	474
query74	9868	9772	9571	9571
query75	3565	3402	3021	3021
query76	2368	1158	757	757
query77	383	411	329	329
query78	11349	11345	10773	10773
query79	3255	1073	802	802
query80	1739	753	643	643
query81	552	276	242	242
query82	920	155	120	120
query83	339	282	262	262
query84	271	143	119	119
query85	932	493	451	451
query86	502	313	356	313
query87	3334	3215	3094	3094
query88	3612	2682	2691	2682
query89	482	419	384	384
query90	1966	175	179	175
query91	179	179	145	145
query92	88	78	76	76
query93	1654	882	502	502
query94	697	290	253	253
query95	667	367	427	367
query96	1025	744	330	330
query97	2693	2702	2551	2551
query98	247	237	232	232
query99	1090	1075	953	953
Total cold run time: 263956 ms
Total hot run time: 180152 ms

jacktengg · 2026-04-03T03:57:50Z

run buildall

jacktengg · 2026-04-03T03:57:57Z

/review

github-actions · 2026-04-03T04:05:09Z

No blocking issues found in this PR.

Critical checkpoint conclusions:

Goal and correctness: The change achieves its stated goal of extending the block fast path to explode_outer and single-argument posexplode while preserving the existing row-wise fallback. I did not find a semantic mismatch in the modified BE path.
Scope and focus: The functional changes stay localized to the table-function fast path and the corresponding fast-path capability checks.
Concurrency: No new concurrent state, lock ordering, or lifecycle synchronization concerns were introduced in this patch.
Special lifecycle / static init: Not applicable.
Config changes: None.
Compatibility / storage / FE-BE protocol: No incompatible serialization, persistence, or FE-BE protocol changes were introduced.
Parallel code paths: The slow path remains intact, and multi-argument vexplode_v2 still stays off the fast path via _multi_detail.size() == 1.
Special conditions: Null/empty-array handling and posexplode struct materialization are handled explicitly in the new fast path, with assertions guarding the contiguity assumption.
Test coverage: Added BE unit coverage for inner explode with null/empty arrays, outer explode with null/empty arrays, and posexplode fast path. Residual risk: I did not see a direct posexplode_outer fast-path unit/regression case in this PR.
Observability: No additional observability appears necessary for this optimization-only change.
Transaction / persistence / data-write safety: Not applicable.
Performance: The segment-based bulk copy approach is a reasonable optimization and removes the old per-row accumulation bottleneck on the covered path.
Other issues: None found during review.

Overall opinion: looks correct and sufficiently contained, with only a minor residual test gap around direct posexplode_outer fast-path coverage.

doris-robot · 2026-04-03T05:05:06Z

TPC-H: Total hot run time: 29032 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3c661d0ed2ad33c60d2b8f2db869a22a32fccd9, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17754	3721	3698	3698
q2	q3	10683	875	597	597
q4	4672	468	367	367
q5	7449	1341	1152	1152
q6	191	170	138	138
q7	925	951	791	791
q8	9305	1448	1337	1337
q9	5590	5293	5313	5293
q10	6280	2050	1812	1812
q11	470	276	281	276
q12	638	413	284	284
q13	18054	2810	2162	2162
q14	289	290	257	257
q15	q16	857	848	795	795
q17	1060	1163	669	669
q18	6391	5711	5575	5575
q19	1150	1301	1101	1101
q20	568	425	286	286
q21	4736	2498	2090	2090
q22	480	483	352	352
Total cold run time: 97542 ms
Total hot run time: 29032 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4635	4443	4629	4443
q2	q3	4610	4791	4217	4217
q4	2087	2169	1335	1335
q5	4912	4932	5230	4932
q6	206	169	138	138
q7	2025	1741	1769	1741
q8	3662	3266	3153	3153
q9	8242	8221	8236	8221
q10	4470	4515	4296	4296
q11	614	401	380	380
q12	681	720	486	486
q13	2724	3353	2383	2383
q14	313	313	276	276
q15	q16	751	776	677	677
q17	1327	1268	1214	1214
q18	7913	7189	7101	7101
q19	1170	1111	1107	1107
q20	2208	2199	1938	1938
q21	5984	5315	4767	4767
q22	545	504	452	452
Total cold run time: 59079 ms
Total hot run time: 53257 ms

doris-robot · 2026-04-03T05:16:23Z

TPC-DS: Total hot run time: 178944 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3c661d0ed2ad33c60d2b8f2db869a22a32fccd9, data reload: false

query5	4374	697	510	510
query6	328	236	203	203
query7	4229	558	346	346
query8	331	250	227	227
query9	8736	3914	3889	3889
query10	451	372	314	314
query11	6710	5496	5104	5104
query12	188	141	131	131
query13	1324	655	465	465
query14	5707	5253	4765	4765
query14_1	4158	4103	4083	4083
query15	207	202	184	184
query16	1018	450	441	441
query17	966	750	628	628
query18	2465	498	373	373
query19	251	212	159	159
query20	134	128	132	128
query21	219	151	123	123
query22	14028	14764	14834	14764
query23	18088	17181	16628	16628
query23_1	16795	16786	16913	16786
query24	7452	1791	1337	1337
query24_1	1340	1357	1349	1349
query25	583	510	432	432
query26	1259	330	174	174
query27	2697	639	375	375
query28	4491	1913	1897	1897
query29	941	670	562	562
query30	294	238	196	196
query31	1099	1047	950	950
query32	85	71	71	71
query33	542	349	285	285
query34	1296	1210	679	679
query35	781	791	664	664
query36	1212	1221	1073	1073
query37	156	101	89	89
query38	3106	3037	2958	2958
query39	930	887	856	856
query39_1	830	838	839	838
query40	234	149	130	130
query41	62	60	56	56
query42	112	108	101	101
query43	312	322	283	283
query44	
query45	208	192	185	185
query46	1154	1242	810	810
query47	2341	2335	2220	2220
query48	406	440	303	303
query49	630	560	433	433
query50	784	298	221	221
query51	4310	4263	4212	4212
query52	110	108	96	96
query53	266	280	208	208
query54	330	278	257	257
query55	96	94	88	88
query56	303	320	319	319
query57	1722	1622	1667	1622
query58	307	274	275	274
query59	2967	3022	2720	2720
query60	337	341	315	315
query61	163	161	177	161
query62	683	635	572	572
query63	251	203	195	195
query64	5267	1399	1069	1069
query65	
query66	1510	478	369	369
query67	24310	24383	24114	24114
query68	
query69	462	352	319	319
query70	987	988	1028	988
query71	320	283	266	266
query72	3101	2835	2461	2461
query73	831	816	469	469
query74	9879	9747	9533	9533
query75	2844	2666	2319	2319
query76	2306	1189	800	800
query77	407	413	334	334
query78	11461	11299	10786	10786
query79	1338	1124	826	826
query80	699	615	534	534
query81	448	294	240	240
query82	254	165	129	129
query83	295	302	267	267
query84	294	164	126	126
query85	917	579	543	543
query86	352	325	329	325
query87	3250	3210	3125	3125
query88	3664	2736	2735	2735
query89	449	398	377	377
query90	2187	198	185	185
query91	197	188	160	160
query92	80	84	71	71
query93	987	996	571	571
query94	537	362	304	304
query95	705	386	334	334
query96	1040	815	370	370
query97	2692	2694	2550	2550
query98	242	233	225	225
query99	1059	1065	958	958
Total cold run time: 257082 ms
Total hot run time: 178944 ms

HappenLee · 2026-04-03T06:25:11Z

/review

github-actions · 2026-04-03T06:32:11Z

No issues found in this review.

Checkpoint conclusions:

Goal and correctness: The patch does accomplish its stated goal of extending the block fast path to explode_outer and posexplode, and the added BE unit tests cover the main inner/outer/null/empty-array cases.
Change scope: The implementation stays focused on the table-function fast path plus targeted tests.
Concurrency: No new concurrent state, locking, or lifecycle hazards were introduced in the touched code path.
Lifecycle/static init: No special lifecycle or static-initialization risk found in the changes reviewed.
Compatibility/config: No config, protocol, storage-format, or rolling-upgrade compatibility changes involved.
Parallel paths: The PR updates both explode and vexplode_v2 fast-path eligibility, so the relevant parallel execution paths appear covered.
Conditional logic: The new boundary handling for null/empty rows and segment flushing is justified by surrounding comments and matches the existing row-wise semantics.
Test coverage: BE unit coverage was substantially expanded for the new fast-path cases. I also attempted to run ./run-be-ut.sh --run table_function_operator_test, but it timed out in environment bootstrap/submodule setup before executing the target test, so I did not get an end-to-end local pass in this runner.
Observability: No new observability requirement stood out for this optimization-only change.
Transaction/persistence/data-write concerns: Not applicable; this PR is limited to BE execution of table functions.
FE/BE variable passing: No new FE-BE transmitted fields beyond internal fast-path context state.
Performance: The single-pass segment accumulation looks directionally sound and avoids the prior row-by-row copying in the covered cases.
Other issues: None identified from the code paths reviewed.

Residual risk: variant-array / broader integration behavior was not exercised locally in this runner because the BE UT command did not finish setup in time.

jacktengg · 2026-04-03T06:54:04Z

run buildall

be/src/exec/operator/table_function_operator.cpp

doris-robot · 2026-04-03T08:10:11Z

TPC-H: Total hot run time: 29242 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5cb8aeaaebcabbfee8bed674d65f3ef637411df6, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17669	3803	3804	3803
q2	q3	10672	860	612	612
q4	4681	476	366	366
q5	7450	1345	1153	1153
q6	188	175	145	145
q7	910	946	779	779
q8	9315	1460	1321	1321
q9	5498	5363	5227	5227
q10	6247	2026	1756	1756
q11	480	275	285	275
q12	632	410	299	299
q13	18043	2797	2191	2191
q14	286	284	258	258
q15	q16	856	862	787	787
q17	1071	1041	873	873
q18	6533	5753	5555	5555
q19	1187	1339	1059	1059
q20	564	432	305	305
q21	4272	2547	2113	2113
q22	508	457	365	365
Total cold run time: 97062 ms
Total hot run time: 29242 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4804	4722	4740	4722
q2	q3	4661	4841	4258	4258
q4	2039	2114	1385	1385
q5	4921	5045	5170	5045
q6	219	175	143	143
q7	2068	1827	1613	1613
q8	3288	3078	3058	3058
q9	7956	8327	8683	8327
q10	4513	4567	4263	4263
q11	605	441	399	399
q12	660	703	534	534
q13	2750	3018	2320	2320
q14	303	351	324	324
q15	q16	852	862	746	746
q17	1404	1335	1298	1298
q18	7947	7157	7298	7157
q19	1125	1129	1128	1128
q20	2263	2270	2033	2033
q21	6189	5472	4827	4827
q22	555	540	425	425
Total cold run time: 59122 ms
Total hot run time: 54005 ms

doris-robot · 2026-04-03T08:21:27Z

TPC-DS: Total hot run time: 178889 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5cb8aeaaebcabbfee8bed674d65f3ef637411df6, data reload: false

query5	4359	695	531	531
query6	345	225	202	202
query7	4275	599	349	349
query8	343	256	233	233
query9	8745	3972	3959	3959
query10	458	364	316	316
query11	6643	5449	5132	5132
query12	194	137	132	132
query13	1296	626	446	446
query14	5731	5213	4805	4805
query14_1	4199	4237	4132	4132
query15	224	206	189	189
query16	1017	460	461	460
query17	958	819	671	671
query18	2449	509	388	388
query19	238	216	176	176
query20	138	136	130	130
query21	230	149	131	131
query22	14108	14788	14629	14629
query23	17914	17429	16590	16590
query23_1	16776	16916	16652	16652
query24	7464	1769	1335	1335
query24_1	1338	1375	1385	1375
query25	580	492	444	444
query26	1265	320	176	176
query27	2704	638	378	378
query28	4483	1915	1890	1890
query29	953	654	539	539
query30	297	228	192	192
query31	1088	1047	935	935
query32	82	69	73	69
query33	551	343	291	291
query34	1202	1179	681	681
query35	758	779	679	679
query36	1234	1211	1121	1121
query37	152	98	88	88
query38	3149	3088	2967	2967
query39	922	907	848	848
query39_1	823	844	830	830
query40	232	146	132	132
query41	62	60	59	59
query42	110	105	105	105
query43	317	320	278	278
query44	
query45	208	195	189	189
query46	1127	1275	790	790
query47	2326	2352	2248	2248
query48	409	413	307	307
query49	650	536	441	441
query50	759	303	223	223
query51	4328	4256	4246	4246
query52	109	113	104	104
query53	255	275	205	205
query54	326	268	256	256
query55	98	92	87	87
query56	307	325	313	313
query57	1699	1674	1543	1543
query58	304	272	269	269
query59	2906	2975	2755	2755
query60	338	331	335	331
query61	160	156	159	156
query62	686	624	576	576
query63	246	198	196	196
query64	5213	1302	1000	1000
query65	
query66	1468	471	376	376
query67	24272	24191	24349	24191
query68	
query69	447	337	305	305
query70	1049	994	979	979
query71	323	286	277	277
query72	2918	2739	2436	2436
query73	845	781	465	465
query74	9820	9742	9566	9566
query75	2739	2630	2307	2307
query76	2302	1148	781	781
query77	407	431	352	352
query78	11350	11350	10774	10774
query79	1510	1154	856	856
query80	709	587	506	506
query81	458	286	239	239
query82	1380	152	119	119
query83	364	290	269	269
query84	319	152	121	121
query85	867	506	441	441
query86	385	365	322	322
query87	3306	3208	3099	3099
query88	3686	2753	2763	2753
query89	435	406	353	353
query90	2009	183	181	181
query91	178	170	145	145
query92	83	74	70	70
query93	987	999	563	563
query94	542	359	314	314
query95	665	386	423	386
query96	1044	823	344	344
query97	2665	2672	2555	2555
query98	245	228	240	228
query99	1082	1053	961	961
Total cold run time: 256949 ms
Total hot run time: 178889 ms

jacktengg · 2026-04-03T08:55:03Z

run buildall

HappenLee

LGTM

github-actions · 2026-04-03T09:14:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-04-03T09:14:52Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

doris-robot · 2026-04-03T10:07:38Z

TPC-H: Total hot run time: 29088 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e8199dbf3eedaa7cc3bbdf0ea9f06fb938827ed, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17593	3697	3736	3697
q2	q3	10690	872	603	603
q4	4675	479	371	371
q5	7450	1335	1158	1158
q6	183	167	137	137
q7	911	946	779	779
q8	9323	1421	1337	1337
q9	5521	5354	5264	5264
q10	6270	2032	1776	1776
q11	473	287	272	272
q12	637	398	281	281
q13	18072	2806	2160	2160
q14	284	284	258	258
q15	q16	865	864	788	788
q17	1046	1064	886	886
q18	6459	5643	5576	5576
q19	1143	1274	1061	1061
q20	550	412	291	291
q21	3936	2648	2032	2032
q22	503	454	361	361
Total cold run time: 96584 ms
Total hot run time: 29088 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4575	4463	4953	4463
q2	q3	4622	4770	4151	4151
q4	2025	2078	1336	1336
q5	4905	4954	5198	4954
q6	190	166	130	130
q7	1957	1733	1595	1595
q8	3281	3214	3244	3214
q9	8270	8255	8378	8255
q10	4486	4542	4286	4286
q11	628	431	380	380
q12	641	725	495	495
q13	2728	3029	2300	2300
q14	287	302	268	268
q15	q16	774	767	678	678
q17	1304	1287	1195	1195
q18	7835	7030	7048	7030
q19	1247	1134	1138	1134
q20	2206	2179	1915	1915
q21	6031	5416	4808	4808
q22	562	507	435	435
Total cold run time: 58554 ms
Total hot run time: 53022 ms

doris-robot · 2026-04-03T10:18:51Z

TPC-DS: Total hot run time: 177616 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e8199dbf3eedaa7cc3bbdf0ea9f06fb938827ed, data reload: false

query5	4354	654	525	525
query6	331	231	207	207
query7	4272	563	350	350
query8	341	251	233	233
query9	8729	3865	3851	3851
query10	458	364	322	322
query11	6721	5501	5142	5142
query12	193	132	128	128
query13	1311	608	461	461
query14	5649	5162	4783	4783
query14_1	4165	4122	4104	4104
query15	219	207	182	182
query16	1001	470	472	470
query17	938	801	660	660
query18	2458	523	403	403
query19	243	215	175	175
query20	142	131	133	131
query21	237	150	122	122
query22	13568	13563	13270	13270
query23	17796	17541	16714	16714
query23_1	16835	16989	16811	16811
query24	7910	1858	1413	1413
query24_1	1412	1421	1458	1421
query25	654	582	448	448
query26	1327	342	201	201
query27	2786	663	401	401
query28	4501	1860	1860	1860
query29	950	672	543	543
query30	297	233	194	194
query31	1087	1038	945	945
query32	85	72	69	69
query33	527	343	279	279
query34	1182	1179	685	685
query35	729	757	662	662
query36	1267	1301	1039	1039
query37	149	97	86	86
query38	3068	3029	2984	2984
query39	904	890	845	845
query39_1	825	836	847	836
query40	229	145	129	129
query41	61	59	61	59
query42	111	108	108	108
query43	312	322	268	268
query44	
query45	201	195	184	184
query46	1164	1217	803	803
query47	2353	2363	2225	2225
query48	426	397	306	306
query49	634	516	442	442
query50	747	290	215	215
query51	4355	4290	4197	4197
query52	107	109	94	94
query53	260	273	196	196
query54	315	267	259	259
query55	104	91	88	88
query56	310	325	316	316
query57	1748	1717	1630	1630
query58	301	271	271	271
query59	2884	2981	2723	2723
query60	346	326	324	324
query61	156	149	157	149
query62	698	624	578	578
query63	237	198	198	198
query64	5281	1349	979	979
query65	
query66	1478	484	369	369
query67	24302	24383	24208	24208
query68	
query69	445	338	304	304
query70	1013	972	971	971
query71	311	289	275	275
query72	3059	2772	2531	2531
query73	857	783	450	450
query74	9879	9753	9546	9546
query75	2741	2627	2315	2315
query76	2323	1129	783	783
query77	395	415	353	353
query78	11317	11423	10768	10768
query79	1493	1112	803	803
query80	692	611	495	495
query81	459	288	236	236
query82	1410	160	125	125
query83	376	292	268	268
query84	310	153	122	122
query85	901	513	458	458
query86	406	341	328	328
query87	3264	3188	3107	3107
query88	3552	2688	2715	2688
query89	436	396	353	353
query90	1998	171	178	171
query91	181	170	142	142
query92	82	76	72	72
query93	965	980	573	573
query94	541	335	319	319
query95	645	452	347	347
query96	1003	795	331	331
query97	2707	2669	2590	2590
query98	244	231	229	229
query99	1069	1072	980	980
Total cold run time: 256923 ms
Total hot run time: 177616 ms

doris-robot · 2026-04-03T10:36:46Z

BE UT Coverage Report

Increment line coverage 88.72% (118/133) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.98% (20076/37895)
Line Coverage	36.56% (188626/515979)
Region Coverage	32.83% (146515/446341)
Branch Coverage	33.95% (64106/188812)

hello-stephen · 2026-04-03T11:14:12Z

BE Regression && UT Coverage Report

Increment line coverage 91.67% (143/156) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.68% (27342/37111)
Line Coverage	57.25% (294515/514398)
Region Coverage	54.51% (245547/450475)
Branch Coverage	56.18% (106410/189394)

hello-stephen · 2026-04-06T08:22:18Z

BE Regression && UT Coverage Report

Increment line coverage 91.67% (143/156) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.68% (27342/37111)
Line Coverage	57.26% (294536/514398)
Region Coverage	54.52% (245580/450475)
Branch Coverage	56.19% (106428/189394)

Issue Number: close #xxx Related PR: #xxx Problem Summary: Improvement of apache#60352, optimize explode_outer, posexplode and posexplode_outer. The approach is a single-pass algorithm: walk through child rows, accumulating contiguous segments into the output, then when hitting a null/empty row or reaching the end, flush the segment using bulk operations. For outer-null rows, insert a NULL and copy the non-table-function columns directly. This naturally handles both outer and non-outer modes since non-outer mode just won't produce any null outputs. For posexplode, generate position indices alongside this. Performance test result. Create and populate a large test table. Use 100,000,000 base rows, each with an array of 10 INT elements, producing 1,000,000,000 total exploded output rows. | Function | slow path | optimized | SpeedUp| | --------- |---------- |-----------|----------| | explode | 9734 ms | 5105 ms |90%| | explode_outer | 9685 ms | 5064 ms |91%| | posexplode | 11021 ms | 5963 ms |84%| | posexplode_outer | 14088 ms | 5903 ms |138%| None - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Improvement of apache#60352, optimize explode_outer, posexplode and posexplode_outer. The approach is a single-pass algorithm: walk through child rows, accumulating contiguous segments into the output, then when hitting a null/empty row or reaching the end, flush the segment using bulk operations. For outer-null rows, insert a NULL and copy the non-table-function columns directly. This naturally handles both outer and non-outer modes since non-outer mode just won't produce any null outputs. For posexplode, generate position indices alongside this. Performance test result. Create and populate a large test table. Use 100,000,000 base rows, each with an array of 10 INT elements, producing 1,000,000,000 total exploded output rows. | Function | slow path | optimized | SpeedUp| | --------- |---------- |-----------|----------| | explode | 9734 ms | 5105 ms |90%| | explode_outer | 9685 ms | 5064 ms |91%| | posexplode | 11021 ms | 5963 ms |84%| | posexplode_outer | 14088 ms | 5903 ms |138%| ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

Issue Number: Pick #62069 Related PR: #xxx Problem Summary: Improvement of #60352, optimize explode_outer, posexplode and posexplode_outer. The approach is a single-pass algorithm: walk through child rows, accumulating contiguous segments into the output, then when hitting a null/empty row or reaching the end, flush the segment using bulk operations. For outer-null rows, insert a NULL and copy the non-table-function columns directly. This naturally handles both outer and non-outer modes since non-outer mode just won't produce any null outputs. For posexplode, generate position indices alongside this. Performance test result. Create and populate a large test table. Use 100,000,000 base rows, each with an array of 10 INT elements, producing 1,000,000,000 total exploded output rows. | Function | slow path | optimized | SpeedUp| | --------- |---------- |-----------|----------| | explode | 9734 ms | 5105 ms |90%| | explode_outer | 9685 ms | 5064 ms |91%| | posexplode | 11021 ms | 5963 ms |84%| | posexplode_outer | 14088 ms | 5903 ms |138%| None - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

[opt](explode) Optimize explode_outer and posexplode

c3c661d

jacktengg force-pushed the 260402-explode-opt branch from 4b0a118 to c3c661d Compare April 3, 2026 03:57

fix beut

5cb8aea

HappenLee reviewed Apr 3, 2026

View reviewed changes

be/src/exec/operator/table_function_operator.cpp Show resolved Hide resolved

HappenLee reviewed Apr 3, 2026

View reviewed changes

be/src/exec/operator/table_function_operator.cpp Outdated Show resolved Hide resolved

improve

6e8199d

jacktengg added the dev/4.1.x label Apr 3, 2026

HappenLee approved these changes Apr 3, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2026

github-actions bot added the reviewed label Apr 3, 2026

eldenmoon approved these changes Apr 3, 2026

View reviewed changes

yiguolei merged commit 157a35e into apache:master Apr 7, 2026
31 of 33 checks passed

github-actions bot added the dev/4.1.x-conflict label Apr 7, 2026

jacktengg mentioned this pull request Apr 7, 2026

[opt](explode) Optimize explode_outer and posexplode (#62069) #62145

Merged

29 tasks

yiguolei added dev/4.1.0-merged and removed dev/4.1.x dev/4.1.x-conflict labels Apr 7, 2026

Conversation

jacktengg commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 3, 2026

Uh oh!

jacktengg commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

jacktengg commented Apr 3, 2026

Uh oh!

jacktengg commented Apr 3, 2026

Uh oh!

github-actions bot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

HappenLee commented Apr 3, 2026

Uh oh!

github-actions bot commented Apr 3, 2026

Uh oh!

jacktengg commented Apr 3, 2026

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

jacktengg commented Apr 3, 2026

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 3, 2026

Uh oh!

github-actions bot commented Apr 3, 2026

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

Uh oh!

doris-robot commented Apr 3, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 3, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 6, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jacktengg commented Apr 3, 2026 •

edited

Loading