[fix](fe) Fix row binlog schema change test metadata by seawinde · Pull Request #65663 · apache/doris

seawinde · 2026-07-15T11:35:56Z

What problem does this PR solve?

Issue Number: N/A

Related PR: #65076

Problem Summary:
The master FE test compilation fails because SchemaChangeHandlerTest
references the removed Column.BINLOG_TIMESTAMP_COL constant. Two
OlapTableRowBinlogSchemaTest assertions also expect LSN indexes from the
previous metadata layout.

Root cause: PR #65076 added these tests using the previous row-binlog metadata
schema. PR #64133 subsequently replaced the timestamp column with TSO,
changed the LSN type to BIGINT, and ordered the metadata columns as
TSO, LSN, and operation before #65076 was merged.

This change aligns both test fixtures with the current production schema,
updates the expected LSN indexes, and explicitly verifies TSO positions.

File	Change Description
`SchemaChangeHandlerTest.java`	Use the current TSO, LSN, and operation metadata columns and update the expected LSN index
`OlapTableRowBinlogSchemaTest.java`	Verify TSO positions and align LSN indexes with the current metadata order

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
  - mvn install -DskipTests -Dmaven.build.cache.enabled=false
  - ./run-fe-ut.sh --run org.apache.doris.catalog.OlapTableRowBinlogSchemaTest,org.apache.doris.alter.SchemaChangeHandlerTest
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: N/A Related PR: apache#65076 Problem Summary: SchemaChangeHandlerTest still referenced the removed BINLOG_TIMESTAMP_COL after row binlog metadata changed to use TSO. Align the test fixture with the current TSO, LSN, and operation columns so master test compilation succeeds. ### Release note None ### Check List (For Author) - Test: Unit Test - mvn install -DskipTests -Dmaven.build.cache.enabled=false - ./run-fe-ut.sh --run org.apache.doris.alter.SchemaChangeHandlerTest - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-07-15T11:36:01Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2026-07-15T11:36:37Z

run buildall

hello-stephen · 2026-07-15T11:53:12Z

TPC-H: Total hot run time: 29065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 291418d52d00be260fe6be84d52381fa8946c11b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17633	4098	4064	4064
q2	2149	324	200	200
q3	10317	1418	807	807
q4	4718	472	337	337
q5	7877	829	567	567
q6	245	168	137	137
q7	761	815	611	611
q8	10412	1432	1548	1432
q9	5991	4337	4277	4277
q10	6834	1712	1435	1435
q11	507	339	313	313
q12	701	558	418	418
q13	18209	3342	2728	2728
q14	275	259	252	252
q15	q16	789	776	707	707
q17	971	908	996	908
q18	7082	5809	5621	5621
q19	2753	1261	1027	1027
q20	790	635	528	528
q21	5944	2590	2409	2409
q22	421	352	287	287
Total cold run time: 105379 ms
Total hot run time: 29065 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4475	4259	4278	4259
q2	296	323	211	211
q3	4574	4934	4391	4391
q4	2066	2141	1365	1365
q5	4386	4242	4265	4242
q6	232	172	127	127
q7	2309	1878	1564	1564
q8	2491	2116	2178	2116
q9	7760	7673	7859	7673
q10	4659	4627	4226	4226
q11	564	402	384	384
q12	943	766	550	550
q13	3349	3558	2970	2970
q14	299	307	269	269
q15	q16	717	735	648	648
q17	1369	1335	1311	1311
q18	8032	7352	6837	6837
q19	1131	1068	1086	1068
q20	2220	2215	1938	1938
q21	5206	4548	4441	4441
q22	515	445	431	431
Total cold run time: 57593 ms
Total hot run time: 51021 ms

hello-stephen · 2026-07-15T11:54:01Z

skip buildall

### What problem does this PR solve? Issue Number: N/A Related PR: apache#65076 Problem Summary: OlapTableRowBinlogSchemaTest expected LSN indexes from the previous row binlog metadata layout. Align the assertions with the current TSO, LSN, and operation order and verify the TSO indexes. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.catalog.OlapTableRowBinlogSchemaTest,org.apache.doris.alter.SchemaChangeHandlerTest - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-07-15T12:02:14Z

run buildall

hello-stephen · 2026-07-15T12:20:52Z

TPC-H: Total hot run time: 29841 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3940f650c70df631b311fa018fb6164500555cd0, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17755	4151	4136	4136
q2	2111	323	209	209
q3	10203	1466	862	862
q4	4718	484	346	346
q5	7740	859	581	581
q6	214	174	136	136
q7	759	813	614	614
q8	9628	1633	1652	1633
q9	5849	4332	4357	4332
q10	6758	1766	1476	1476
q11	517	359	320	320
q12	706	560	437	437
q13	18089	3489	2835	2835
q14	274	260	243	243
q15	q16	795	781	715	715
q17	975	891	989	891
q18	7217	5786	5684	5684
q19	1160	1301	1087	1087
q20	757	660	539	539
q21	5553	2676	2463	2463
q22	440	363	302	302
Total cold run time: 102218 ms
Total hot run time: 29841 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4372	4301	4338	4301
q2	293	326	221	221
q3	4616	4978	4473	4473
q4	2070	2144	1386	1386
q5	4462	4307	4275	4275
q6	228	173	128	128
q7	1753	1661	1577	1577
q8	2315	2034	1975	1975
q9	7329	7274	7236	7236
q10	4631	4582	4133	4133
q11	562	426	405	405
q12	756	758	537	537
q13	3018	3417	2816	2816
q14	288	289	259	259
q15	q16	694	707	629	629
q17	1310	1267	1277	1267
q18	7441	6901	6834	6834
q19	1068	1083	1081	1081
q20	2251	2226	1944	1944
q21	5368	4657	4574	4574
q22	539	499	428	428
Total cold run time: 55364 ms
Total hot run time: 50479 ms

hello-stephen · 2026-07-15T12:31:53Z

TPC-DS: Total hot run time: 177029 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3940f650c70df631b311fa018fb6164500555cd0, data reload: false

query5	4147	615	503	503
query6	472	230	195	195
query7	4876	599	328	328
query8	347	199	167	167
query9	8762	4107	4101	4101
query10	451	357	306	306
query11	6034	2337	2152	2152
query12	182	113	104	104
query13	1308	594	431	431
query14	5873	5220	4895	4895
query14_1	4329	4255	4233	4233
query15	210	207	179	179
query16	616	500	444	444
query17	1059	740	601	601
query18	487	446	351	351
query19	280	190	144	144
query20	109	104	102	102
query21	175	159	134	134
query22	13589	13596	13416	13416
query23	17429	16501	16168	16168
query23_1	16237	16136	16120	16120
query24	7491	1749	1264	1264
query24_1	1286	1298	1261	1261
query25	599	423	371	371
query26	1191	343	203	203
query27	2511	563	381	381
query28	4360	2007	2007	2007
query29	1031	587	476	476
query30	359	261	225	225
query31	1116	1080	970	970
query32	115	68	66	66
query33	520	306	253	253
query34	1165	1135	647	647
query35	774	781	690	690
query36	1190	1190	1051	1051
query37	148	108	88	88
query38	1905	1712	1665	1665
query39	894	868	836	836
query39_1	831	833	827	827
query40	249	156	137	137
query41	64	62	63	62
query42	91	88	89	88
query43	321	328	286	286
query44	1403	773	757	757
query45	195	191	168	168
query46	1095	1177	695	695
query47	2123	2142	2024	2024
query48	399	413	302	302
query49	573	421	310	310
query50	1020	407	322	322
query51	10767	10690	10448	10448
query52	88	89	85	85
query53	254	285	192	192
query54	287	230	208	208
query55	75	68	64	64
query56	302	302	273	273
query57	1316	1278	1188	1188
query58	282	246	254	246
query59	1596	1732	1467	1467
query60	293	275	252	252
query61	151	143	151	143
query62	568	498	437	437
query63	244	201	203	201
query64	2778	1077	854	854
query65	4713	4604	4571	4571
query66	1885	493	375	375
query67	29458	29286	28521	28521
query68	3115	1556	1039	1039
query69	437	303	264	264
query70	1088	954	900	900
query71	384	320	302	302
query72	3063	2675	2569	2569
query73	849	791	441	441
query74	5080	4983	4720	4720
query75	2541	2499	2138	2138
query76	2339	1206	773	773
query77	372	382	293	293
query78	11988	11906	11174	11174
query79	1347	1120	744	744
query80	683	575	488	488
query81	477	332	280	280
query82	609	167	124	124
query83	413	326	296	296
query84	296	164	138	138
query85	1057	670	636	636
query86	372	282	297	282
query87	1824	1843	1775	1775
query88	3716	2862	2770	2770
query89	421	365	327	327
query90	1852	197	194	194
query91	194	190	163	163
query92	64	56	55	55
query93	1532	1449	922	922
query94	545	348	325	325
query95	789	510	565	510
query96	1021	825	353	353
query97	2630	2641	2500	2500
query98	214	213	202	202
query99	1087	1103	979	979
Total cold run time: 254031 ms
Total hot run time: 177029 ms

shuke987 · 2026-07-15T12:33:43Z

skip buildall

hello-stephen · 2026-07-15T12:36:50Z

ClickBench: Total hot run time: 25.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3940f650c70df631b311fa018fb6164500555cd0, data reload: false

query1	0.00	0.00	0.01
query2	0.10	0.05	0.05
query3	0.26	0.14	0.14
query4	1.62	0.14	0.14
query5	0.24	0.25	0.24
query6	1.29	1.11	1.11
query7	0.05	0.01	0.01
query8	0.06	0.03	0.04
query9	0.40	0.32	0.32
query10	0.57	0.59	0.54
query11	0.21	0.15	0.14
query12	0.19	0.14	0.15
query13	0.48	0.47	0.46
query14	1.02	1.04	1.03
query15	0.62	0.60	0.60
query16	0.32	0.33	0.32
query17	1.15	1.12	1.09
query18	0.22	0.20	0.21
query19	2.07	1.94	1.88
query20	0.02	0.01	0.02
query21	15.42	0.18	0.14
query22	5.00	0.05	0.05
query23	16.10	0.31	0.12
query24	3.03	0.42	0.35
query25	0.10	0.06	0.05
query26	0.77	0.22	0.14
query27	0.05	0.03	0.04
query28	3.49	0.91	0.54
query29	12.51	4.10	3.29
query30	0.28	0.16	0.15
query31	2.77	0.60	0.32
query32	3.22	0.59	0.48
query33	3.21	3.28	3.23
query34	15.70	4.24	3.52
query35	3.55	3.55	3.58
query36	0.58	0.43	0.41
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.06	0.04	0.04
Total cold run time: 97.21 s
Total hot run time: 25.2 s

github-actions · 2026-07-15T12:39:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-07-15T12:39:24Z

PR approved by anyone and no changes requested.

hello-stephen · 2026-07-15T13:20:28Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Issue Number: N/A Related PR: apache#65076, apache#65663 Problem Summary: The row-binlog writer BE unit test no longer compiles after allocate_binlog_lsn changed to accept a vector reference. After that compile error is fixed, the test aborts because its filtered row-binlog schema retains the metadata indexes from before two hidden columns were removed and its assertions still expect the previous LSN, operation, and timestamp layout. Initialize the LSN vector before passing it by reference, reset the TSO, LSN, and operation indexes after filtering, and verify values using the current metadata order and types. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run '--filter=GroupRowsetWriterTest.*' - Behavior changed: No - Does this need documentation: No

Related PR: #65076, #65663 Problem Summary: The master BE unit test target fails to compile because `GroupRowsetWriterTest` still passes a pointer to `allocate_binlog_lsn()`, which now accepts a vector reference. After fixing compilation, the suite aborts while flushing row-binlog data. Root cause: In `GroupRowsetWriterTest.SetUp()`, two columns are removed from the generated row-binlog schema without updating `binlog_tso_idx`, `binlog_lsn_idx`, and `binlog_op_idx`. `RowBinlogSegmentWriter` consequently accesses column ID 7 in a seven-column schema. The test output assertions also use the previous LSN, operation, and timestamp layout. This change initializes the shared LSN vector before passing it by reference, resets the metadata indexes after filtering columns, and verifies TSO, LSN, and operation values using the current schema order and types.

### What problem does this PR solve? Issue Number: N/A Related PR: apache#65076, apache#65663 Problem Summary: The row-binlog writer BE unit test no longer compiles after allocate_binlog_lsn changed to accept a vector reference. After that compile error is fixed, the test aborts because its filtered row-binlog schema retains the metadata indexes from before two hidden columns were removed and its assertions still expect the previous LSN, operation, and timestamp layout. Initialize the LSN vector before passing it by reference, reset the TSO, LSN, and operation indexes after filtering, and verify values using the current metadata order and types. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run '--filter=GroupRowsetWriterTest.*' - Behavior changed: No - Does this need documentation: No

* [fix](row-binlog) Support IVM MTMV binlog access ### What problem does this PR solve? Issue Number: N/A Related PR: #20 Problem Summary: IVM MTMVs use a hidden row-id key that is persisted in the row-binlog schema. The generic binlog table function rejected MTMV tables and could expose internal IVM columns unconditionally. Allow OlapTable subclasses such as MTMV, hide IVM internal columns by default, expose them when show_hidden_columns is enabled, and add focused BE and regression coverage. The latest yujun777/ivm base already contains the generic hidden-key writer support from upstream, so this commit reuses that implementation instead of replacing it. ### Release note Support querying row binlog for incremental materialized views while keeping internal IVM columns hidden by default. ### Check List (For Author) - Test: Not run yet (verification follows after applying the related BE test compatibility change) - Behavior changed: Yes. binlog() accepts IVM MTMV tables and follows show_hidden_columns for internal IVM columns. - Does this need documentation: No * [fix](be) Fix row binlog writer test metadata ### What problem does this PR solve? Issue Number: N/A Related PR: apache#65076, apache#65663 Problem Summary: The row-binlog writer BE unit test no longer compiles after allocate_binlog_lsn changed to accept a vector reference. After that compile error is fixed, the test aborts because its filtered row-binlog schema retains the metadata indexes from before two hidden columns were removed and its assertions still expect the previous LSN, operation, and timestamp layout. Initialize the LSN vector before passing it by reference, reset the TSO, LSN, and operation indexes after filtering, and verify values using the current metadata order and types. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run '--filter=GroupRowsetWriterTest.*' - Behavior changed: No - Does this need documentation: No

seawinde requested review from gavinchou and luwei16 as code owners July 15, 2026 11:35

luwei16 approved these changes Jul 15, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2026

github-actions Bot added the reviewed label Jul 15, 2026

hello-stephen merged commit 52729f1 into apache:master Jul 15, 2026
33 checks passed

seawinde mentioned this pull request Jul 16, 2026

[fix](be) Fix row binlog writer test metadata #65686

Merged

16 tasks

Uh oh!

Conversation

seawinde commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

seawinde commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

shuke987 commented Jul 15, 2026

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

Uh oh!

hello-stephen commented Jul 15, 2026

FE UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seawinde commented Jul 15, 2026 •

edited

Loading