[opt](fe) opt the performance for setColumnPositionMapping #60003

cambyzju · 2026-01-18T14:26:41Z

What problem does this PR solve?

When there are large of columns(such as 3W columns) in table, setColumnPositionMapping function will be the main performance bottleneck.

this pr opt it from $O(N \times M)$ to $O(N + M)$

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-01-18T14:26:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

cambyzju · 2026-01-18T14:27:12Z

run buildall

doris-robot · 2026-01-18T14:49:28Z

TPC-H: Total hot run time: 32032 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

------ Round 1 ----------------------------------
q1	17631	4278	4042	4042
q2	2041	364	255	255
q3	10133	1280	710	710
q4	10206	811	305	305
q5	7543	2061	1820	1820
q6	188	170	137	137
q7	897	775	667	667
q8	9292	1393	1228	1228
q9	4893	4559	4667	4559
q10	6718	1782	1403	1403
q11	533	298	266	266
q12	696	740	566	566
q13	17791	3807	3112	3112
q14	295	298	272	272
q15	586	503	507	503
q16	680	700	629	629
q17	650	792	487	487
q18	6617	6496	6715	6496
q19	1362	1099	678	678
q20	443	393	252	252
q21	3214	2595	2623	2595
q22	1165	1110	1050	1050
Total cold run time: 103574 ms
Total hot run time: 32032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4264	4358	4264
q2	339	430	340	340
q3	2296	2856	2465	2465
q4	1506	1899	1493	1493
q5	4532	4325	4270	4270
q6	211	169	130	130
q7	2027	2028	1705	1705
q8	2769	2378	2370	2370
q9	7324	7419	7138	7138
q10	2478	2681	2271	2271
q11	534	475	454	454
q12	681	741	611	611
q13	3609	4010	3100	3100
q14	276	281	254	254
q15	519	489	481	481
q16	614	648	610	610
q17	1091	1228	1306	1228
q18	7532	7331	7209	7209
q19	840	787	795	787
q20	1880	2017	1792	1792
q21	4537	4238	4108	4108
q22	1063	1031	1000	1000
Total cold run time: 51098 ms
Total hot run time: 48080 ms

doris-robot · 2026-01-18T15:00:16Z

TPC-DS: Total hot run time: 174521 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

query5	5014	610	489	489
query6	331	221	215	215
query7	4221	452	269	269
query8	328	242	233	233
query9	8751	2890	2907	2890
query10	556	376	314	314
query11	15358	15298	14915	14915
query12	188	111	114	111
query13	1239	502	381	381
query14	6481	3036	2832	2832
query14_1	2691	2636	2705	2636
query15	221	194	174	174
query16	1009	488	484	484
query17	1059	658	566	566
query18	2703	430	343	343
query19	227	217	201	201
query20	126	116	115	115
query21	221	140	119	119
query22	4235	4056	4230	4056
query23	16075	15671	15337	15337
query23_1	15562	15596	15421	15421
query24	7124	1578	1169	1169
query24_1	1178	1194	1191	1191
query25	544	452	430	430
query26	1245	264	151	151
query27	2744	443	278	278
query28	4566	2190	2191	2190
query29	748	516	409	409
query30	309	237	205	205
query31	819	628	577	577
query32	79	74	69	69
query33	533	346	310	310
query34	911	879	529	529
query35	727	748	678	678
query36	878	914	835	835
query37	133	100	84	84
query38	2700	2721	2666	2666
query39	770	745	736	736
query39_1	744	717	703	703
query40	208	134	118	118
query41	67	62	65	62
query42	111	105	103	103
query43	440	428	414	414
query44	1320	750	746	746
query45	182	184	180	180
query46	818	934	566	566
query47	1430	1483	1333	1333
query48	323	318	247	247
query49	607	453	351	351
query50	627	283	208	208
query51	3892	3790	3867	3790
query52	101	105	94	94
query53	285	324	279	279
query54	287	265	259	259
query55	80	78	78	78
query56	305	291	311	291
query57	1051	962	888	888
query58	268	257	269	257
query59	1992	2117	1968	1968
query60	337	338	317	317
query61	150	146	146	146
query62	380	363	311	311
query63	300	263	267	263
query64	4824	1237	954	954
query65	3843	3837	3775	3775
query66	1388	424	307	307
query67	15639	15557	15396	15396
query68	2430	1095	783	783
query69	424	367	323	323
query70	989	871	917	871
query71	330	312	302	302
query72	5292	3313	3364	3313
query73	617	720	316	316
query74	8821	8825	8658	8658
query75	2763	2835	2471	2471
query76	2304	1058	648	648
query77	367	391	315	315
query78	9772	9993	9159	9159
query79	1077	924	600	600
query80	1465	591	537	537
query81	520	270	232	232
query82	1352	146	112	112
query83	362	250	244	244
query84	250	122	94	94
query85	1020	484	430	430
query86	389	302	289	289
query87	2922	2842	2778	2778
query88	3467	2584	2561	2561
query89	405	344	322	322
query90	1928	173	164	164
query91	166	168	133	133
query92	76	76	73	73
query93	941	891	553	553
query94	617	313	303	303
query95	599	398	320	320
query96	633	504	232	232
query97	2349	2432	2328	2328
query98	212	206	202	202
query99	597	588	545	545
Total cold run time: 249173 ms
Total hot run time: 174521 ms

doris-robot · 2026-01-18T15:05:16Z

ClickBench: Total hot run time: 26.89 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.26	0.09	0.09
query4	1.60	0.12	0.11
query5	0.28	0.26	0.26
query6	1.15	0.67	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.59	0.59
query14	0.95	0.95	0.93
query15	0.80	0.78	0.78
query16	0.38	0.43	0.39
query17	1.02	1.02	1.04
query18	0.22	0.21	0.21
query19	1.94	1.90	1.84
query20	0.02	0.03	0.01
query21	15.47	0.29	0.15
query22	4.98	0.06	0.05
query23	15.75	0.28	0.10
query24	1.58	0.23	0.25
query25	0.10	0.05	0.05
query26	0.14	0.14	0.14
query27	0.08	0.09	0.08
query28	3.60	1.09	0.88
query29	12.54	3.93	3.18
query30	0.28	0.14	0.12
query31	2.81	0.63	0.40
query32	3.25	0.56	0.47
query33	3.02	3.05	3.03
query34	16.55	5.09	4.46
query35	4.53	4.43	4.47
query36	0.65	0.50	0.48
query37	0.10	0.07	0.06
query38	0.07	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.73 s
Total hot run time: 26.89 s

hello-stephen · 2026-01-18T15:38:19Z

FE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉
Increment coverage report
Complete coverage report

cambyzju · 2026-01-19T02:18:01Z

run p0

cambyzju · 2026-01-19T02:56:51Z

run cloud_p0

cambyzju · 2026-01-19T02:56:58Z

run external

hello-stephen · 2026-01-19T04:07:55Z

FE Regression Coverage Report

Increment line coverage 83.33% (5/6) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-19T04:37:54Z

FE Regression Coverage Report

Increment line coverage 83.33% (5/6) 🎉
Increment coverage report
Complete coverage report

cambyzju · 2026-01-19T06:53:39Z

run buildall

doris-robot · 2026-01-19T07:27:47Z

TPC-H: Total hot run time: 31305 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f4c976b92545a9ee0dd8ac7eebbc6ba6c76241e, data reload: false

------ Round 1 ----------------------------------
q1	17648	4187	4061	4061
q2	2035	346	237	237
q3	10154	1259	718	718
q4	10192	789	303	303
q5	7538	2029	1861	1861
q6	191	171	135	135
q7	944	760	650	650
q8	9295	1361	1154	1154
q9	4925	4663	4529	4529
q10	6743	1810	1378	1378
q11	547	311	293	293
q12	701	745	594	594
q13	17772	3862	3087	3087
q14	306	303	278	278
q15	574	498	505	498
q16	685	674	646	646
q17	647	799	496	496
q18	6583	6510	6373	6373
q19	1098	969	636	636
q20	398	362	248	248
q21	3004	2416	2180	2180
q22	1031	986	950	950
Total cold run time: 103011 ms
Total hot run time: 31305 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4008	4061	4008
q2	347	395	312	312
q3	2114	2598	2239	2239
q4	1325	1745	1329	1329
q5	4105	3942	4080	3942
q6	206	169	126	126
q7	1861	1827	1669	1669
q8	2802	2538	2452	2452
q9	7189	7228	7284	7228
q10	2507	2938	2351	2351
q11	562	487	458	458
q12	707	764	665	665
q13	3821	4116	3346	3346
q14	299	316	292	292
q15	639	561	515	515
q16	628	681	652	652
q17	1141	1342	1393	1342
q18	8075	7528	7882	7528
q19	861	812	849	812
q20	2077	2117	1956	1956
q21	4829	4461	4268	4268
q22	1168	1098	1038	1038
Total cold run time: 51340 ms
Total hot run time: 48528 ms

github-actions · 2026-01-19T07:33:40Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-01-19T07:33:43Z

PR approved by anyone and no changes requested.

doris-robot · 2026-01-19T07:38:35Z

TPC-DS: Total hot run time: 174991 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f4c976b92545a9ee0dd8ac7eebbc6ba6c76241e, data reload: false

query5	4398	636	514	514
query6	335	235	211	211
query7	4216	447	271	271
query8	331	248	242	242
query9	8743	2927	2885	2885
query10	520	378	327	327
query11	15343	15331	15941	15331
query12	184	132	120	120
query13	1293	553	442	442
query14	6043	3141	2950	2950
query14_1	2799	2848	2714	2714
query15	210	204	183	183
query16	1079	511	515	511
query17	1106	750	614	614
query18	2490	443	342	342
query19	227	222	204	204
query20	120	113	117	113
query21	204	132	111	111
query22	3810	4270	3934	3934
query23	16168	15496	15213	15213
query23_1	15496	15358	15545	15358
query24	7119	1539	1178	1178
query24_1	1168	1173	1163	1163
query25	554	458	403	403
query26	1244	260	154	154
query27	2755	441	281	281
query28	4522	2193	2225	2193
query29	753	505	411	411
query30	298	234	207	207
query31	808	646	549	549
query32	79	72	71	71
query33	515	338	308	308
query34	917	859	528	528
query35	715	751	676	676
query36	892	887	796	796
query37	123	96	79	79
query38	2730	2841	2719	2719
query39	775	752	746	746
query39_1	705	713	711	711
query40	218	131	118	118
query41	63	60	60	60
query42	101	100	104	100
query43	485	440	407	407
query44	1324	734	731	731
query45	187	176	178	176
query46	823	954	561	561
query47	1437	1484	1307	1307
query48	318	321	263	263
query49	596	442	341	341
query50	615	272	215	215
query51	3766	3820	3772	3772
query52	104	107	95	95
query53	304	320	273	273
query54	298	276	271	271
query55	86	82	78	78
query56	305	305	316	305
query57	1017	1013	945	945
query58	269	259	258	258
query59	2101	2128	2168	2128
query60	372	334	322	322
query61	154	143	149	143
query62	380	356	301	301
query63	298	260	263	260
query64	4922	1252	987	987
query65	3828	3696	3727	3696
query66	1451	434	305	305
query67	15517	15525	15544	15525
query68	2384	1102	773	773
query69	448	379	344	344
query70	1023	967	961	961
query71	349	323	300	300
query72	5308	3242	3224	3224
query73	598	726	319	319
query74	8753	8671	8661	8661
query75	2741	2782	2484	2484
query76	2260	1052	678	678
query77	369	390	304	304
query78	9702	9981	9101	9101
query79	1407	902	593	593
query80	1291	572	485	485
query81	543	265	233	233
query82	965	160	110	110
query83	329	255	241	241
query84	254	121	90	90
query85	909	486	443	443
query86	414	293	301	293
query87	2833	2831	2748	2748
query88	3527	2619	2568	2568
query89	400	350	323	323
query90	2017	170	163	163
query91	165	162	136	136
query92	76	75	75	75
query93	1095	892	551	551
query94	637	347	285	285
query95	585	332	320	320
query96	647	515	230	230
query97	2353	2390	2301	2301
query98	229	199	195	195
query99	590	575	498	498
Total cold run time: 247471 ms
Total hot run time: 174991 ms

doris-robot · 2026-01-19T07:43:38Z

ClickBench: Total hot run time: 26.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7f4c976b92545a9ee0dd8ac7eebbc6ba6c76241e, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.04	0.04
query3	0.26	0.08	0.09
query4	1.63	0.11	0.11
query5	0.27	0.26	0.26
query6	1.14	0.67	0.65
query7	0.03	0.03	0.02
query8	0.06	0.04	0.04
query9	0.58	0.51	0.50
query10	0.55	0.57	0.54
query11	0.15	0.10	0.10
query12	0.15	0.10	0.11
query13	0.60	0.60	0.59
query14	0.95	0.94	0.95
query15	0.79	0.78	0.78
query16	0.40	0.39	0.43
query17	1.01	1.07	1.00
query18	0.22	0.21	0.21
query19	1.97	1.88	1.87
query20	0.02	0.01	0.02
query21	15.64	0.29	0.14
query22	5.99	0.05	0.05
query23	15.99	0.31	0.10
query24	1.54	0.73	0.18
query25	0.10	0.10	0.05
query26	0.14	0.14	0.13
query27	0.05	0.08	0.05
query28	4.58	1.06	0.89
query29	12.60	4.00	3.13
query30	0.28	0.14	0.11
query31	2.81	0.65	0.39
query32	3.25	0.56	0.46
query33	2.99	3.07	3.06
query34	16.08	5.09	4.51
query35	4.47	4.44	4.42
query36	0.65	0.49	0.48
query37	0.10	0.07	0.06
query38	0.07	0.04	0.03
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.65 s
Total hot run time: 26.76 s

hello-stephen · 2026-01-19T08:03:06Z

FE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-19T09:47:48Z

FE Regression Coverage Report

Increment line coverage 83.33% (5/6) 🎉
Increment coverage report
Complete coverage report

morningman

LGTM

### What problem does this PR solve? When there are large of columns(such as 3W columns) in table, setColumnPositionMapping function will be the main performance bottleneck. this pr opt it from $O(N \times M)$ to $O(N + M)$

…#60003 (#60037) Cherry-picked from #60003 Co-authored-by: camby <104178625@qq.com>

### What problem does this PR solve? like #60003, When there are large of columns(such as 3W columns) in table, TupleDescriptor.getSlot function will be the main performance bottleneck. **Before** ``` - Init Scan Node Time: 1s94ms ``` **After** ``` - Init Scan Node Time: 145ms ``` ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

opt setColumnPositionMapping

3429f97

Merge branch 'master' into opt-master-20250118

1966db5

use first column index for duplicate column names

7f4c976

liutang123 approved these changes Jan 19, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 19, 2026

github-actions bot added the reviewed label Jan 19, 2026

cambyzju added dev/3.1.x dev/4.0.x dev/4.1.x labels Jan 19, 2026

morningman self-assigned this Jan 19, 2026

morningman approved these changes Jan 19, 2026

View reviewed changes

morningman merged commit a2733f0 into apache:master Jan 19, 2026
35 of 36 checks passed

github-actions bot mentioned this pull request Jan 19, 2026

branch-4.0: [opt](fe) opt the performance for setColumnPositionMapping #60003 #60037

Merged

yiguolei pushed a commit that referenced this pull request Jan 20, 2026

branch-4.0: [opt](fe) opt the performance for setColumnPositionMapping …

0e458a3

…#60003 (#60037) Cherry-picked from #60003 Co-authored-by: camby <104178625@qq.com>

yiguolei added dev/4.0.3-merged and removed dev/4.0.x dev/4.1.x labels Jan 20, 2026

cambyzju mentioned this pull request Jan 21, 2026

[opt](fe) opt the performance for TupleDescriptor.getSlot #60115

Merged

16 tasks

[opt](fe) opt the performance for setColumnPositionMapping #60003

[opt](fe) opt the performance for setColumnPositionMapping #60003

Uh oh!

Conversation

cambyzju commented Jan 18, 2026 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 18, 2026

Uh oh!

cambyzju commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

doris-robot commented Jan 18, 2026

Uh oh!

hello-stephen commented Jan 18, 2026

FE UT Coverage Report

Uh oh!

cambyzju commented Jan 19, 2026

Uh oh!

cambyzju commented Jan 19, 2026

Uh oh!

cambyzju commented Jan 19, 2026

Uh oh!

hello-stephen commented Jan 19, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Jan 19, 2026

FE Regression Coverage Report

Uh oh!

cambyzju commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

doris-robot commented Jan 19, 2026

Uh oh!

hello-stephen commented Jan 19, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 19, 2026

FE Regression Coverage Report

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cambyzju commented Jan 18, 2026 •

edited by morningman

Loading