Skip to content

[opt](column) avoid redundant column clones#65367

Open
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:opt-remove-redundant-column-clone
Open

[opt](column) avoid redundant column clones#65367
Mryange wants to merge 1 commit into
apache:masterfrom
Mryange:opt-remove-redundant-column-clone

Conversation

@Mryange

@Mryange Mryange commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Some column execution paths copied full column data even when the result could safely share existing column storage and rely on COW before mutation. Root cause: projection, partition filtering, array offsets, and null-condition IF handling used clone_resized() as a defensive ownership boundary.

This change removes those redundant clones by reusing expression result columns directly, sharing array offsets in array_map, filtering copied blocks through the existing COW-aware block filter path, and reusing the else column when an IF condition is entirely null.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange

Mryange commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 10th, 2026 6:44 AM.
Workflow run: https://github.com/apache/doris/actions/runs/28930207906

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Mryange

Mryange commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 9.09% (2/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.54% (23209/41051)
Line Coverage 40.10% (225218/561655)
Region Coverage 36.06% (177923/493460)
Branch Coverage 37.10% (79067/213099)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (11/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.04% (29267/40071)
Line Coverage 56.62% (316189/558438)
Region Coverage 53.03% (262781/495546)
Branch Coverage 54.15% (115512/213334)

@Mryange

Mryange commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29490 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef525f924dc654e1d9c366b73b7b5fa3d7098200, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17597	4087	4029	4029
q2	2033	326	200	200
q3	10316	1455	825	825
q4	4684	473	339	339
q5	7508	883	570	570
q6	190	173	140	140
q7	791	851	629	629
q8	9322	1534	1612	1534
q9	5709	4423	4393	4393
q10	6731	1783	1533	1533
q11	513	351	316	316
q12	707	549	435	435
q13	18112	3435	2766	2766
q14	277	262	252	252
q15	q16	795	788	709	709
q17	1027	955	1024	955
q18	6960	5719	5503	5503
q19	1297	1270	1066	1066
q20	789	653	542	542
q21	5772	2569	2452	2452
q22	436	362	302	302
Total cold run time: 101566 ms
Total hot run time: 29490 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4432	4311	4299	4299
q2	294	313	208	208
q3	4594	4964	4455	4455
q4	2014	2167	1366	1366
q5	4442	4385	4389	4385
q6	236	186	132	132
q7	1719	1906	1923	1906
q8	2534	2243	2191	2191
q9	7945	8210	7896	7896
q10	4726	4700	4269	4269
q11	593	443	394	394
q12	734	774	571	571
q13	3296	3545	3024	3024
q14	330	288	279	279
q15	q16	734	740	639	639
q17	1344	1303	1291	1291
q18	8155	7445	7174	7174
q19	1136	1124	1129	1124
q20	2197	2193	1923	1923
q21	5250	4556	4437	4437
q22	541	447	424	424
Total cold run time: 57246 ms
Total hot run time: 52387 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 179189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef525f924dc654e1d9c366b73b7b5fa3d7098200, data reload: false

query5	4333	675	502	502
query6	469	246	225	225
query7	4862	620	334	334
query8	342	197	182	182
query9	8792	4145	4105	4105
query10	482	384	298	298
query11	5937	2371	2150	2150
query12	166	105	105	105
query13	1290	622	457	457
query14	6270	5375	4998	4998
query14_1	4346	4382	4345	4345
query15	224	215	183	183
query16	1022	488	465	465
query17	967	752	607	607
query18	2476	489	360	360
query19	213	197	163	163
query20	115	109	108	108
query21	232	158	143	143
query22	13646	13622	13369	13369
query23	17555	16651	16162	16162
query23_1	16301	16392	16317	16317
query24	7633	1811	1329	1329
query24_1	1354	1321	1341	1321
query25	585	468	402	402
query26	1400	363	215	215
query27	2577	597	393	393
query28	4481	2108	2041	2041
query29	1048	628	477	477
query30	340	257	225	225
query31	1114	1107	975	975
query32	102	61	60	60
query33	525	310	252	252
query34	1179	1186	663	663
query35	780	768	653	653
query36	1388	1382	1239	1239
query37	154	108	93	93
query38	1878	1703	1640	1640
query39	931	921	889	889
query39_1	883	905	875	875
query40	249	162	143	143
query41	68	66	65	65
query42	94	95	93	93
query43	343	348	295	295
query44	1511	798	788	788
query45	204	197	181	181
query46	1139	1250	744	744
query47	2327	2299	2161	2161
query48	389	434	302	302
query49	583	448	323	323
query50	1019	428	328	328
query51	10666	10852	10441	10441
query52	89	89	76	76
query53	265	282	203	203
query54	278	242	233	233
query55	77	72	67	67
query56	303	298	292	292
query57	1414	1413	1305	1305
query58	291	250	254	250
query59	1649	1694	1488	1488
query60	316	272	258	258
query61	155	150	180	150
query62	687	624	579	579
query63	251	212	209	209
query64	2814	1072	878	878
query65	4896	4757	4758	4757
query66	1817	515	406	406
query67	28771	28654	28409	28409
query68	3255	1465	1025	1025
query69	405	311	271	271
query70	1077	963	972	963
query71	349	333	322	322
query72	3130	2302	2456	2302
query73	892	758	462	462
query74	5120	5044	4762	4762
query75	2623	2574	2210	2210
query76	2323	1233	787	787
query77	396	388	281	281
query78	12313	12450	11663	11663
query79	1436	1192	762	762
query80	1277	540	467	467
query81	518	329	268	268
query82	557	153	122	122
query83	379	318	297	297
query84	329	159	128	128
query85	968	599	526	526
query86	464	311	300	300
query87	1852	1825	1770	1770
query88	3771	2823	2791	2791
query89	471	409	369	369
query90	1923	208	200	200
query91	198	195	156	156
query92	65	64	56	56
query93	1704	1542	974	974
query94	846	375	324	324
query95	780	505	469	469
query96	1061	791	364	364
query97	2691	2676	2550	2550
query98	215	210	203	203
query99	1127	1167	1015	1015
Total cold run time: 265977 ms
Total hot run time: 179189 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef525f924dc654e1d9c366b73b7b5fa3d7098200, data reload: false

query1	0.01	0.00	0.01
query2	0.10	0.05	0.05
query3	0.25	0.14	0.14
query4	1.60	0.14	0.15
query5	0.28	0.23	0.24
query6	1.29	1.07	1.04
query7	0.04	0.00	0.01
query8	0.06	0.04	0.04
query9	0.38	0.32	0.32
query10	0.59	0.55	0.54
query11	0.19	0.15	0.14
query12	0.18	0.14	0.14
query13	0.46	0.47	0.48
query14	1.02	1.02	1.00
query15	0.62	0.60	0.60
query16	0.33	0.31	0.34
query17	1.11	1.09	1.12
query18	0.23	0.21	0.22
query19	2.13	1.96	1.98
query20	0.02	0.02	0.01
query21	15.44	0.23	0.13
query22	4.73	0.05	0.06
query23	16.16	0.31	0.12
query24	3.05	0.42	0.32
query25	0.13	0.04	0.04
query26	0.72	0.21	0.14
query27	0.03	0.03	0.04
query28	3.50	0.93	0.54
query29	12.48	4.07	3.24
query30	0.27	0.16	0.15
query31	2.79	0.59	0.32
query32	3.22	0.60	0.48
query33	3.15	3.16	3.21
query34	15.75	4.27	3.54
query35	3.60	3.49	3.53
query36	0.55	0.43	0.45
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.99 s
Total hot run time: 24.99 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 9.09% (2/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.54% (23209/41051)
Line Coverage 40.10% (225238/561655)
Region Coverage 36.06% (177932/493460)
Branch Coverage 37.11% (79075/213099)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (11/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.04% (29267/40071)
Line Coverage 56.62% (316200/558438)
Region Coverage 53.04% (262828/495546)
Branch Coverage 54.15% (115516/213334)

1 similar comment
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (11/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.04% (29267/40071)
Line Coverage 56.62% (316200/558438)
Region Coverage 53.04% (262828/495546)
Branch Coverage 54.15% (115516/213334)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (11/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.05% (29271/40071)
Line Coverage 56.64% (316272/558438)
Region Coverage 53.04% (262862/495546)
Branch Coverage 54.16% (115542/213334)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 50.00% (11/22) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.10% (29291/40071)
Line Coverage 56.68% (316514/558438)
Region Coverage 53.06% (262936/495546)
Branch Coverage 54.23% (115695/213334)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants