Skip to content

[Draft](be) Unify storage reader column ordinals - #66472

Open
csun5285 wants to merge 1 commit into
apache:masterfrom
csun5285:refactor/reader-return-columns
Open

[Draft](be) Unify storage reader column ordinals#66472
csun5285 wants to merge 1 commit into
apache:masterfrom
csun5285:refactor/reader-return-columns

Conversation

@csun5285

@csun5285 csun5285 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
csun5285 force-pushed the refactor/reader-return-columns branch from e96085d to 6a3b439 Compare August 7, 2026 12:14
@csun5285
[refactor](be) Unify storage reader column ordinals
768a009 
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#66432

Problem Summary: Storage readers maintained FE block positions, tablet-schema column IDs, predicate IDs, and delete-predicate columns through parallel mappings. Projection, nested-column pruning, virtual expressions, and row-binlog dependencies could make those mappings diverge. Use one ordered read schema as the reader coordinate, keep expected materialization types alongside physical columns, append storage-only delete-predicate dependencies after FE slots, and preserve required row-binlog scan columns without changing the scan output contract.

### Release note

Fix inconsistent storage-reader column mappings for projected and row-binlog scans.

### Check List (For Author)

- Test:
    - ASAN BE and FE build
    - BE clang-format and format check
    - FE Checkstyle
    - Regression: row_binlog_p0, delete_p0, unique_seq_map_p0, and targeted schema-change/delete/sequence suites
    - Regression: variant_p0 code-related cases passed; one outfile case was blocked by invalid external OSS credentials
- Behavior changed: Yes (reader column identity and row-binlog dependency handling are unified)
- Does this need documentation: No
@csun5285
csun5285 force-pushed the refactor/reader-return-columns branch from 6a3b439 to 768a009 Compare August 7, 2026 12:27
@csun5285

csun5285 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29131 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 768a0090d6d77fef5a40d4a42f3daad744c933f7, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17707	4084	3969	3969
q2	2013	334	195	195
q3	10307	1380	799	799
q4	4683	464	339	339
q5	7510	852	553	553
q6	186	163	141	141
q7	742	808	611	611
q8	9335	1411	1477	1411
q9	5307	4089	4061	4061
q10	6798	1623	1360	1360
q11	504	366	324	324
q12	735	565	439	439
q13	18078	3670	2730	2730
q14	264	266	253	253
q15	q16	730	726	658	658
q17	1001	1077	1009	1009
q18	6508	5615	5534	5534
q19	1178	1250	1153	1153
q20	789	676	558	558
q21	6133	2876	2725	2725
q22	457	381	309	309
Total cold run time: 100965 ms
Total hot run time: 29131 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4927	4543	4590	4543
q2	295	320	207	207
q3	4880	5209	4737	4737
q4	2176	2256	1400	1400
q5	4496	4541	4391	4391
q6	229	186	146	146
q7	1888	1726	1497	1497
q8	2333	2014	1989	1989
q9	7135	6825	6660	6660
q10	4261	4223	3809	3809
q11	512	367	335	335
q12	688	694	492	492
q13	2953	3320	2737	2737
q14	265	278	249	249
q15	q16	672	679	618	618
q17	1221	1208	1199	1199
q18	12174	10958	11683	10958
q19	1105	1065	1084	1065
q20	2186	2190	1900	1900
q21	5170	4605	4522	4522
q22	512	446	414	414
Total cold run time: 60078 ms
Total hot run time: 53868 ms

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 79.00% (79/100) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 158453 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 768a0090d6d77fef5a40d4a42f3daad744c933f7, data reload: false

query5	4319	579	452	452
query6	462	220	227	220
query7	4856	613	345	345
query8	321	169	160	160
query9	8773	3988	3960	3960
query10	452	366	299	299
query11	5847	2202	2033	2033
query12	149	96	94	94
query13	1254	588	423	423
query14	6072	4241	3979	3979
query14_1	3775	3747	3772	3747
query15	206	197	176	176
query16	1004	469	447	447
query17	960	682	529	529
query18	2428	454	321	321
query19	201	180	139	139
query20	105	100	102	100
query21	234	155	135	135
query22	13043	12972	12782	12782
query23	15908	15121	14563	14563
query23_1	14728	14730	14685	14685
query24	7467	1700	1226	1226
query24_1	1253	1219	1199	1199
query25	524	466	356	356
query26	1303	353	216	216
query27	2587	628	375	375
query28	4554	1986	2005	1986
query29	1053	586	467	467
query30	338	260	222	222
query31	1179	1117	1052	1052
query32	109	61	61	61
query33	527	298	234	234
query34	1214	1159	649	649
query35	721	748	624	624
query36	769	772	675	675
query37	159	103	88	88
query38	1831	1746	1663	1663
query39	827	827	803	803
query39_1	794	781	784	781
query40	246	166	145	145
query41	66	71	64	64
query42	98	95	90	90
query43	320	321	277	277
query44	1412	759	760	759
query45	186	179	180	179
query46	1070	1161	743	743
query47	1572	1557	1446	1446
query48	404	396	298	298
query49	573	412	295	295
query50	1041	436	336	336
query51	10747	10767	10807	10767
query52	88	88	82	82
query53	274	280	201	201
query54	303	259	233	233
query55	78	80	69	69
query56	331	324	301	301
query57	1041	975	930	930
query58	316	282	261	261
query59	1539	1601	1372	1372
query60	322	291	275	275
query61	179	183	177	177
query62	395	327	273	273
query63	241	194	200	194
query64	3012	1179	978	978
query65	3900	3817	3793	3793
query66	1842	498	380	380
query67	20056	20028	19936	19936
query68	3392	1642	1051	1051
query69	431	303	274	274
query70	910	799	772	772
query71	370	354	332	332
query72	3232	2827	2320	2320
query73	824	750	459	459
query74	4632	4481	4281	4281
query75	2366	2344	1983	1983
query76	2423	1155	747	747
query77	339	356	277	277
query78	11182	11120	10552	10552
query79	1438	1195	772	772
query80	663	532	469	469
query81	488	338	293	293
query82	629	166	130	130
query83	409	326	306	306
query84	325	163	129	129
query85	938	612	524	524
query86	315	229	216	216
query87	1961	1950	1828	1828
query88	3688	2793	2764	2764
query89	387	316	279	279
query90	1939	190	189	189
query91	202	185	168	168
query92	64	65	56	56
query93	1568	1575	947	947
query94	535	338	311	311
query95	774	488	481	481
query96	1058	834	353	353
query97	2447	2452	2332	2332
query98	196	183	180	180
query99	753	718	605	605
Total cold run time: 244954 ms
Total hot run time: 158453 ms

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 768a0090d6d77fef5a40d4a42f3daad744c933f7, data reload: false

query1	0.00	0.00	0.00
query2	0.09	0.04	0.04
query3	0.25	0.13	0.13
query4	1.62	0.14	0.13
query5	0.23	0.21	0.22
query6	1.17	0.81	0.79
query7	0.04	0.01	0.01
query8	0.06	0.03	0.04
query9	0.37	0.30	0.33
query10	0.57	0.58	0.60
query11	0.18	0.14	0.13
query12	0.18	0.14	0.14
query13	0.46	0.46	0.48
query14	1.00	1.00	0.98
query15	0.60	0.60	0.58
query16	0.33	0.32	0.31
query17	1.14	1.10	1.08
query18	0.22	0.19	0.20
query19	1.98	1.98	1.97
query20	0.02	0.01	0.01
query21	15.44	0.20	0.13
query22	4.96	0.05	0.05
query23	16.36	0.30	0.12
query24	2.95	0.42	0.31
query25	0.12	0.04	0.04
query26	0.72	0.21	0.14
query27	0.04	0.03	0.03
query28	3.57	0.76	0.37
query29	12.52	4.08	3.22
query30	0.27	0.16	0.16
query31	2.76	0.56	0.31
query32	3.22	0.59	0.48
query33	3.18	3.21	3.19
query34	15.58	3.94	3.24
query35	3.27	3.21	3.23
query36	0.54	0.43	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.05	0.04	0.03
Total cold run time: 96.53 s
Total hot run time: 23.85 s

@hello-stephen

hello-stephen commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 84.00% (84/100) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner
@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner
@eldenmoon eldenmoon Awaiting requested review from eldenmoon eldenmoon is a code owner
@airborne12 airborne12 Awaiting requested review from airborne12 airborne12 is a code owner
@luwei16 luwei16 Awaiting requested review from luwei16 luwei16 is a code owner
@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01 liaoxin01 is a code owner
@deardeng deardeng Awaiting requested review from deardeng deardeng is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@csun5285 @hello-stephen