Skip to content

[refine](expr) Migrate old VExpr execute interface to new execute_column (Part 1)#61912

Merged
BiteTheDDDDt merged 3 commits intoapache:masterfrom
Mryange:expr-api-ref-part-1
Mar 31, 2026
Merged

[refine](expr) Migrate old VExpr execute interface to new execute_column (Part 1)#61912
BiteTheDDDDt merged 3 commits intoapache:masterfrom
Mryange:expr-api-ref-part-1

Conversation

@Mryange
Copy link
Copy Markdown
Contributor

@Mryange Mryange commented Mar 31, 2026

What problem does this PR solve?

Changes

File Description
exec/operator/hashjoin_build_sink.cpp ASOF join build-side expr: use ColumnPtr directly, remove result_col_idx and the subsequent block->get_by_position indirection
exec/operator/join/process_hash_table_probe_impl.h ASOF join probe-side expr: same pattern as above
exec/sink/vrow_distribution.cpp Where-clause filter: remove result_index tracking and the block->erase cleanup loop (no longer needed with new interface)
storage/segment/segment_iterator.cpp Virtual column materialization: remove prev_block_columns tracking and erase_tail call; fix stale result_cid reference in log message
format/transformer/iceberg_partition_function.cpp Iceberg insert/delete partition hash: change vector<int> results to vector<ColumnWithTypeAndName> results, use ColumnWithTypeAndName overload
exprs/table_function/vjson_each.cpp VExpr::execute (3-arg) → execute_column
exprs/table_function/vexplode_numbers.cpp Same as above
exprs/table_function/vexplode_map.cpp Same as above; also fix error message to use _collection_column->get_name() instead of stale block position lookup
exec/operator/operator.cpp Bug fix: Remove dead result_column_id = -1 that was never updated after migration to new interface; bytes_usage was never accumulated due to if (result_column_id >= origin_columns_count) always being false

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange
Copy link
Copy Markdown
Contributor Author

Mryange commented Mar 31, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 26687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b92b63ab3ed2e0ae826d08c2504091e6abd9725, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17660	4355	4257	4257
q2	q3	10648	790	531	531
q4	4675	356	249	249
q5	7572	1246	1024	1024
q6	174	171	145	145
q7	806	854	681	681
q8	9899	1477	1373	1373
q9	5130	4708	4760	4708
q10	6321	1903	1663	1663
q11	477	265	252	252
q12	741	577	461	461
q13	18043	2700	1928	1928
q14	233	226	204	204
q15	q16	732	731	656	656
q17	720	842	459	459
q18	5860	5445	5322	5322
q19	1446	983	612	612
q20	534	496	371	371
q21	4595	1837	1510	1510
q22	462	373	281	281
Total cold run time: 96728 ms
Total hot run time: 26687 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4849	4718	4618	4618
q2	q3	3896	4431	3829	3829
q4	867	1206	787	787
q5	4049	4423	4323	4323
q6	178	182	140	140
q7	1792	1658	1520	1520
q8	2503	2757	2967	2757
q9	7618	7346	7469	7346
q10	3807	4118	3661	3661
q11	492	441	430	430
q12	490	596	441	441
q13	2507	3057	2025	2025
q14	266	290	275	275
q15	q16	740	818	733	733
q17	1185	1391	1416	1391
q18	7290	6907	6573	6573
q19	896	861	946	861
q20	2054	2127	2006	2006
q21	3906	3510	3317	3317
q22	462	423	370	370
Total cold run time: 49847 ms
Total hot run time: 47403 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 168011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b92b63ab3ed2e0ae826d08c2504091e6abd9725, data reload: false

query5	4353	662	500	500
query6	331	218	206	206
query7	4221	470	263	263
query8	344	242	220	220
query9	8703	2698	2640	2640
query10	539	381	330	330
query11	6963	5138	4897	4897
query12	188	128	123	123
query13	1279	453	347	347
query14	5782	3721	3472	3472
query14_1	2848	2809	2837	2809
query15	201	194	176	176
query16	977	476	458	458
query17	908	730	663	663
query18	2450	435	339	339
query19	212	203	181	181
query20	136	128	122	122
query21	209	128	109	109
query22	13203	13413	13144	13144
query23	16222	15829	16545	15829
query23_1	16405	16316	16126	16126
query24	7970	1721	1268	1268
query24_1	1301	1274	1253	1253
query25	578	532	422	422
query26	1288	271	155	155
query27	2840	497	313	313
query28	4845	1855	1838	1838
query29	832	591	498	498
query30	308	225	193	193
query31	1024	957	883	883
query32	88	73	74	73
query33	520	349	307	307
query34	906	887	520	520
query35	652	693	599	599
query36	1092	1117	935	935
query37	134	97	85	85
query38	2915	2891	2940	2891
query39	865	850	823	823
query39_1	805	800	789	789
query40	235	158	143	143
query41	69	64	65	64
query42	258	260	256	256
query43	237	246	226	226
query44	
query45	198	194	190	190
query46	877	1001	601	601
query47	3015	2138	2091	2091
query48	319	319	234	234
query49	645	476	396	396
query50	693	282	217	217
query51	4072	4067	4044	4044
query52	263	269	252	252
query53	298	335	287	287
query54	319	286	276	276
query55	95	86	84	84
query56	336	337	326	326
query57	1937	1710	1550	1550
query58	298	281	272	272
query59	2821	2954	2755	2755
query60	354	349	386	349
query61	151	144	147	144
query62	611	583	546	546
query63	305	283	278	278
query64	5066	1270	981	981
query65	
query66	1456	466	362	362
query67	24271	24250	24162	24162
query68	
query69	410	312	284	284
query70	986	946	914	914
query71	332	296	293	293
query72	2784	2687	2417	2417
query73	566	556	315	315
query74	9591	9563	9391	9391
query75	2834	2739	2448	2448
query76	2301	1043	677	677
query77	367	386	318	318
query78	10884	11149	10428	10428
query79	2572	774	580	580
query80	1721	622	545	545
query81	552	257	225	225
query82	1005	151	117	117
query83	330	263	238	238
query84	303	121	103	103
query85	911	510	452	452
query86	418	312	299	299
query87	3136	3181	2988	2988
query88	3507	2681	2617	2617
query89	424	370	340	340
query90	2031	179	172	172
query91	162	171	146	146
query92	79	79	69	69
query93	1065	838	505	505
query94	644	309	287	287
query95	577	404	314	314
query96	636	514	228	228
query97	2462	2479	2447	2447
query98	257	222	222	222
query99	984	1019	905	905
Total cold run time: 253360 ms
Total hot run time: 168011 ms

@Mryange
Copy link
Copy Markdown
Contributor Author

Mryange commented Mar 31, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 26612 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 840f4653cd5b72b36d46b605147c4edef1ad9b4f, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17646	4538	4273	4273
q2	q3	10649	774	522	522
q4	4678	359	256	256
q5	7555	1219	1002	1002
q6	171	173	148	148
q7	769	850	672	672
q8	9317	1486	1358	1358
q9	4980	4819	4748	4748
q10	6293	1936	1638	1638
q11	445	247	252	247
q12	743	572	466	466
q13	18031	2675	1934	1934
q14	229	228	208	208
q15	q16	739	737	651	651
q17	728	828	462	462
q18	6048	5450	5205	5205
q19	1108	981	647	647
q20	546	513	394	394
q21	4561	1857	1449	1449
q22	384	497	332	332
Total cold run time: 95620 ms
Total hot run time: 26612 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4649	4644	4639	4639
q2	q3	3939	4474	3946	3946
q4	909	1217	815	815
q5	4061	4533	4343	4343
q6	198	182	145	145
q7	1762	1647	1547	1547
q8	2525	2685	2567	2567
q9	7608	7495	7370	7370
q10	3733	3968	3627	3627
q11	516	433	425	425
q12	486	602	471	471
q13	2480	3013	2060	2060
q14	310	299	285	285
q15	q16	715	861	739	739
q17	1184	1417	1338	1338
q18	7208	6913	6687	6687
q19	1155	934	935	934
q20	2144	2206	2009	2009
q21	3949	3421	3331	3331
q22	473	511	413	413
Total cold run time: 50004 ms
Total hot run time: 47691 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 168842 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 840f4653cd5b72b36d46b605147c4edef1ad9b4f, data reload: false

query5	4318	633	523	523
query6	338	235	210	210
query7	4202	464	275	275
query8	364	253	228	228
query9	8744	2730	2739	2730
query10	523	388	360	360
query11	7011	5084	4884	4884
query12	178	129	125	125
query13	1263	442	338	338
query14	5549	3640	3438	3438
query14_1	2781	2795	2774	2774
query15	196	191	173	173
query16	969	473	438	438
query17	880	713	609	609
query18	2453	469	363	363
query19	239	215	188	188
query20	136	130	123	123
query21	213	136	116	116
query22	13294	13878	14443	13878
query23	16919	16254	16060	16060
query23_1	16639	16088	15739	15739
query24	7193	1606	1223	1223
query24_1	1235	1244	1231	1231
query25	588	501	442	442
query26	1250	271	154	154
query27	2770	482	299	299
query28	4537	1876	1879	1876
query29	857	600	521	521
query30	306	233	194	194
query31	1005	966	887	887
query32	86	74	72	72
query33	525	363	314	314
query34	918	864	520	520
query35	640	694	612	612
query36	1100	1160	1028	1028
query37	140	98	85	85
query38	2927	3004	2915	2915
query39	852	852	814	814
query39_1	799	802	799	799
query40	235	157	138	138
query41	71	65	64	64
query42	264	257	304	257
query43	239	249	225	225
query44	
query45	192	183	184	183
query46	878	978	605	605
query47	2906	2143	2052	2052
query48	332	313	223	223
query49	630	452	384	384
query50	698	281	217	217
query51	4092	3985	4029	3985
query52	261	268	254	254
query53	297	339	284	284
query54	313	274	264	264
query55	89	90	85	85
query56	311	321	313	313
query57	1908	1794	1565	1565
query58	282	278	277	277
query59	2774	2948	2755	2755
query60	338	339	319	319
query61	157	146	154	146
query62	639	598	555	555
query63	316	282	274	274
query64	5033	1333	1015	1015
query65	
query66	1472	462	363	363
query67	24199	24194	24358	24194
query68	
query69	408	317	303	303
query70	940	957	889	889
query71	338	307	301	301
query72	2816	2669	2449	2449
query73	537	536	321	321
query74	9623	9568	9410	9410
query75	2884	2760	2468	2468
query76	2307	1020	676	676
query77	396	385	312	312
query78	10891	11067	10462	10462
query79	2720	750	585	585
query80	1731	614	550	550
query81	554	266	226	226
query82	999	153	118	118
query83	329	270	243	243
query84	306	125	100	100
query85	936	523	452	452
query86	415	309	312	309
query87	3108	3109	3007	3007
query88	3605	2682	2672	2672
query89	420	366	339	339
query90	2012	186	181	181
query91	174	167	142	142
query92	79	76	71	71
query93	1208	828	502	502
query94	649	310	323	310
query95	589	407	321	321
query96	654	517	224	224
query97	2490	2463	2376	2376
query98	246	221	224	221
query99	1012	981	907	907
Total cold run time: 253053 ms
Total hot run time: 168842 ms

@Mryange
Copy link
Copy Markdown
Contributor Author

Mryange commented Mar 31, 2026

run cloud_p0

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.88% (19947/37720)
Line Coverage 36.43% (187107/513662)
Region Coverage 32.68% (145112/444039)
Branch Coverage 33.84% (63588/187885)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.51% (27152/36937)
Line Coverage 57.08% (292290/512112)
Region Coverage 54.28% (243243/448156)
Branch Coverage 56.05% (105632/188451)

Copy link
Copy Markdown
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2026
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.51% (27152/36937)
Line Coverage 57.08% (292290/512112)
Region Coverage 54.28% (243243/448156)
Branch Coverage 56.05% (105632/188451)

@BiteTheDDDDt BiteTheDDDDt merged commit 01fd754 into apache:master Mar 31, 2026
28 of 30 checks passed
morningman pushed a commit that referenced this pull request Apr 2, 2026
…umn (Part 1) (#61912)

| File | Description |
|------|-------------|
| `exec/operator/hashjoin_build_sink.cpp` | ASOF join build-side expr:
use `ColumnPtr` directly, remove `result_col_idx` and the subsequent
`block->get_by_position` indirection |
| `exec/operator/join/process_hash_table_probe_impl.h` | ASOF join
probe-side expr: same pattern as above |
| `exec/sink/vrow_distribution.cpp` | Where-clause filter: remove
`result_index` tracking and the `block->erase` cleanup loop (no longer
needed with new interface) |
| `storage/segment/segment_iterator.cpp` | Virtual column
materialization: remove `prev_block_columns` tracking and `erase_tail`
call; fix stale `result_cid` reference in log message |
| `format/transformer/iceberg_partition_function.cpp` | Iceberg
insert/delete partition hash: change `vector<int> results` to
`vector<ColumnWithTypeAndName> results`, use `ColumnWithTypeAndName`
overload |
| `exprs/table_function/vjson_each.cpp` | `VExpr::execute` (3-arg) →
`execute_column` |
| `exprs/table_function/vexplode_numbers.cpp` | Same as above |
| `exprs/table_function/vexplode_map.cpp` | Same as above; also fix
error message to use `_collection_column->get_name()` instead of stale
block position lookup |
| `exec/operator/operator.cpp` | **Bug fix**: Remove dead
`result_column_id = -1` that was never updated after migration to new
interface; `bytes_usage` was never accumulated due to `if
(result_column_id >= origin_columns_count)` always being false |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants