Skip to content

[feature](row-binlog) Downlink explicit column mappings - #67456

Open
TsukiokaKogane wants to merge 3 commits into
apache:masterfrom
TsukiokaKogane:fix_resolve_before_column
Open

[feature](row-binlog) Downlink explicit column mappings#67456
TsukiokaKogane wants to merge 3 commits into
apache:masterfrom
TsukiokaKogane:fix_resolve_before_column

Conversation

@TsukiokaKogane

@TsukiokaKogane TsukiokaKogane commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Row-binlog and table-stream paths derived current, before-image, and system-column positions independently in BE from column names and physical layout. This coupled reads and writes to layout details and caused key-only tables requesting historical values to select plain derivation. Compute stable mapping identifiers in FE, downlink them through Thrift and protobuf, resolve ordinals once in BE, and consume explicit CID mappings in write derivation. Preserve mappings only while pending transactions may recover, then clear them before local or Cloud rowsets become visible.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

gavinchou
gavinchou reviewed Sep 2, 2026
View reviewed changes
Comment thread gensrc/proto/olap_file.proto Outdated
// Valid only when segments_overlap_pb is NONOVERLAPPING_WITHIN_GROUP.
// Each value is the number of consecutive output segments in one non-overlapping group.
repeated int32 segment_group_sizes = 59;
optional PRowBinlogWriteColumnMappings row_binlog_column_mappings = 59;

@gavinchou gavinchou Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to persist column mapping in rowset meta, because it is only used in a context that FE can always generate a plan with the column mapping.

@TsukiokaKogane TsukiokaKogane Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's used for write, preserve mappings only while pending transactions may recover (re-publish after restart BE)

@TsukiokaKogane TsukiokaKogane Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里主要是因为 mow表 写带before的binlog是在 publish 阶段执行 update_delete_bitmap的时候做的 这个时候的schema需要是写入时的 所以现在相当于是会在commit 到 publish中间的阶段会临时存一下这个mapping

gavinchou
gavinchou reviewed Sep 2, 2026
View reviewed changes
Comment thread fe/pom.xml Outdated
@TsukiokaKogane

TsukiokaKogane commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@TsukiokaKogane
[feature](row-binlog) Downlink explicit column mappings
b3eaf8a
@TsukiokaKogane
[chore](build) Restore Maven Shade 3.6.2
ff50222 
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: N/A\n\nProblem Summary: Restore the repository's Maven Shade plugin version and comment after the temporary 3.5.1 downgrade, keeping the row-binlog change independent from local build-environment adjustments.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: No need to test (restores the existing Maven Shade configuration)\n- Behavior changed: No\n- Does this need documentation: No
@TsukiokaKogane
[refactor](row-binlog) Centralize special read ordinal resolution
d437321 
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: Row-binlog scanner and merger call sites independently translated TabletSchema TSO, LSN, and OP column ids into dense ReadSchema ordinals. Move that translation into ReadSchema so all read paths resolve special columns through the same TabletSchema unique-id mapping while preserving scan-mode validation at the caller.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ReadSchemaTest.*, RowBinlogVmergeCompactionTest.*, and BlockReaderBinlogVCollectMergeTest.* (14 tests)
- Behavior changed: No
- Does this need documentation: No
@TsukiokaKogane
TsukiokaKogane force-pushed the fix_resolve_before_column branch from 1c803de to d437321 Compare September 3, 2026 03:01
@TsukiokaKogane

TsukiokaKogane commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.71% (2053/2642)
Line Coverage 65.72% (37513/57083)
Region Coverage 52.83% (34898/66057)
Branch Coverage 56.30% (11227/19942)

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d43732131935f056979b6245ac10d4382480078b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17600	3093	3063	3063
q2	2126	269	236	236
q3	10119	873	510	510
q4	4665	253	200	200
q5	7677	566	386	386
q6	138	110	92	92
q7	525	508	403	403
q8	9237	868	912	868
q9	3470	2385	2372	2372
q10	6513	861	716	716
q11	405	196	181	181
q12	626	255	199	199
q13	18128	1514	1193	1193
q14	167	157	137	137
q15	q16	451	392	372	372
q17	1330	890	769	769
q18	3081	2224	2205	2205
q19	1143	932	785	785
q20	375	284	201	201
q21	5241	1627	1826	1627
q22	323	266	232	232
Total cold run time: 93340 ms
Total hot run time: 16747 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3455	3418	3365	3365
q2	496	392	374	374
q3	2240	2330	2118	2118
q4	1174	1156	898	898
q5	2211	2116	2104	2104
q6	170	117	86	86
q7	1040	930	894	894
q8	1594	1414	1410	1410
q9	3100	3070	3048	3048
q10	1857	1775	1600	1600
q11	358	271	258	258
q12	454	425	343	343
q13	1458	1508	1135	1135
q14	164	168	171	168
q15	q16	391	403	366	366
q17	3558	3220	3211	3211
q18	4770	4386	4654	4386
q19	819	772	806	772
q20	1144	970	845	845
q21	3870	3136	3343	3136
q22	394	340	312	312
Total cold run time: 34717 ms
Total hot run time: 30829 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d43732131935f056979b6245ac10d4382480078b, data reload: false

query5	4299	419	340	340
query6	384	138	126	126
query7	4947	417	227	227
query8	298	120	132	120
query9	8713	2893	2895	2893
query10	390	224	185	185
query11	5386	1041	934	934
query12	129	74	74	74
query13	1215	462	329	329
query14	6155	2208	2082	2082
query14_1	1952	1951	1929	1929
query15	182	117	111	111
query16	915	377	369	369
query17	1056	465	375	375
query18	2360	324	239	239
query19	166	162	117	117
query20	85	69	79	69
query21	219	103	94	94
query22	5393	5250	5332	5250
query23	6744	6215	5959	5959
query23_1	6137	6016	6145	6016
query24	7216	1116	764	764
query24_1	778	793	780	780
query25	438	327	262	262
query26	1237	236	133	133
query27	2786	424	259	259
query28	4601	1518	1520	1518
query29	946	441	356	356
query30	250	155	130	130
query31	853	408	330	330
query32	127	74	74	74
query33	470	220	190	190
query34	1011	854	490	490
query35	406	407	351	351
query36	564	585	534	534
query37	120	81	69	69
query38	1032	843	833	833
query39	483	502	481	481
query39_1	452	453	456	453
query40	203	128	75	75
query41	54	51	53	51
query42	72	68	69	68
query43	238	242	215	215
query44	1002	544	550	544
query45	109	102	97	97
query46	788	832	535	535
query47	754	757	728	728
query48	314	305	230	230
query49	529	237	180	180
query50	760	271	191	191
query51	7979	7933	8093	7933
query52	66	65	66	65
query53	204	194	141	141
query54	240	187	161	161
query55	72	58	55	55
query56	196	181	172	172
query57	676	713	664	664
query58	188	166	158	158
query59	1221	1240	1093	1093
query60	237	186	168	168
query61	116	120	125	120
query62	356	223	193	193
query63	170	138	138	138
query64	2682	724	640	640
query65	1601	1613	1559	1559
query66	1802	249	207	207
query67	9798	9786	9675	9675
query68	2998	1196	744	744
query69	349	223	194	194
query70	665	628	606	606
query71	255	186	163	163
query72	2386	1730	1576	1576
query73	664	596	323	323
query74	1996	1210	1140	1140
query75	1179	1093	964	964
query76	2329	719	526	526
query77	260	241	224	224
query78	4005	3588	3192	3192
query79	2790	781	586	586
query80	1603	330	280	280
query81	514	153	130	130
query82	614	119	96	96
query83	271	218	185	185
query84	258	108	87	87
query85	852	342	297	297
query86	482	181	175	175
query87	1019	963	895	895
query88	3050	2111	2107	2107
query89	281	206	178	178
query90	2217	132	126	126
query91	136	122	105	105
query92	105	73	70	70
query93	2421	1089	764	764
query94	653	235	209	209
query95	520	334	232	232
query96	781	559	264	264
query97	1098	1028	993	993
query98	178	134	133	133
query99	434	353	316	316
Total cold run time: 180090 ms
Total hot run time: 81939 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d43732131935f056979b6245ac10d4382480078b, data reload: false

query1	0.01	0.01	0.01
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.61	0.10	0.10
query5	0.17	0.15	0.16
query6	1.25	0.68	0.68
query7	0.03	0.00	0.00
query8	0.06	0.03	0.03
query9	0.30	0.21	0.21
query10	0.34	0.36	0.35
query11	0.18	0.11	0.11
query12	0.14	0.12	0.11
query13	0.29	0.30	0.31
query14	0.46	0.45	0.44
query15	0.36	0.36	0.34
query16	0.23	0.22	0.22
query17	0.66	0.71	0.70
query18	0.18	0.18	0.17
query19	1.23	1.21	1.13
query20	0.02	0.01	0.01
query21	15.45	0.16	0.13
query22	5.02	0.04	0.05
query23	16.17	0.25	0.10
query24	3.05	0.34	0.23
query25	0.10	0.03	0.04
query26	0.72	0.16	0.11
query27	0.03	0.03	0.03
query28	3.59	0.57	0.26
query29	12.47	3.22	2.55
query30	0.28	0.11	0.12
query31	2.76	0.39	0.18
query32	3.51	0.32	0.22
query33	1.48	1.51	1.52
query34	15.36	2.19	1.77
query35	1.76	1.73	1.71
query36	0.48	0.29	0.28
query37	0.07	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 90.49 s
Total hot run time: 14.7 s

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 77.19% (44/57) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou left review comments
@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner
@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01 liaoxin01 is a code owner
@luwei16 luwei16 Awaiting requested review from luwei16 luwei16 is a code owner
@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@eldenmoon eldenmoon Awaiting requested review from eldenmoon eldenmoon is a code owner
@csun5285 csun5285 Awaiting requested review from csun5285 csun5285 is a code owner
@airborne12 airborne12 Awaiting requested review from airborne12 airborne12 is a code owner
@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs
@morningman morningman Awaiting requested review from morningman

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@TsukiokaKogane @hello-stephen @gavinchou