Skip to content

[fix](ccr) Disallow Row Binlog on MOW tables with cluster keys - #66093

Merged
csun5285 merged 1 commit into
apache:masterfrom
csun5285:forbid-row-binlog-cluster-key
Jul 28, 2026
Merged

[fix](ccr) Disallow Row Binlog on MOW tables with cluster keys#66093
csun5285 merged 1 commit into
apache:masterfrom
csun5285:forbid-row-binlog-cluster-key

Conversation

@csun5285

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title [fix](fe) Disallow Row Binlog on MOW tables with cluster keys [fix](binlog) Disallow Row Binlog on MOW tables with cluster keys Jul 27, 2026
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@morrySnow morrySnow changed the title [fix](binlog) Disallow Row Binlog on MOW tables with cluster keys [fix](ccr) Disallow Row Binlog on MOW tables with cluster keys Jul 27, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: Request changes

Findings

  • Preserve the inherited-binlog test database after execution so assertion failures leave the effective table metadata available for debugging.
  • Record the deterministic property and SHOW CREATE results through named regression cases with an auto-generated .out artifact.
  • Hardcode the two fixed ordinary table names at their SQL/query use sites.

Critical checkpoint conclusions

  • Goal and behavior: the FE implementation correctly rejects explicit ROW-binlog MOW cluster keys and prevents the testing-only random-key injection when ROW binlog is effective through database inheritance. The regression covers both behaviors.
  • Scope and clarity: the production change is focused; the only requested changes are in the new regression suite.
  • Concurrency: database binlog state is copied under the database read lock; property updates use the write lock. The analysis/execution double snapshot can fail closed under a concurrent database-property change but cannot publish the unsupported state.
  • Lifecycle and configuration: no new runtime configuration or special object lifecycle is introduced. setFeConfigTemporary restores the existing mutable test config, but the suite's onFinish callback removes diagnostic state even after failure.
  • Compatibility and parallel paths: CREATE, CTAS, and CREATE LIKE converge on the guarded create path; ALTER cannot introduce the combination; ROW-binlog reorder is rejected as a non-light schema change. The replay-only candidate was dismissed because ROW binlog is experimental, master-only, and present in no release tag or release branch.
  • Error handling, persistence, protocol, and data writes: the new invariant raises AnalysisException and is converted to DDL failure before row-binlog metadata or tablets are created. No new persisted field, FE-BE protocol value, transaction path, or data-write path is added.
  • Performance and observability: the only added catalog lookup/copy in analysis is behind the testing-only random-key config; no material hot-path or observability issue was found.
  • Tests: the negative case uses the required test { sql; exception } form and the property query is ordered, but the new suite violates the mandatory preserved-debug-state, generated-oracle, and hardcoded-table-name rules described in the inline comments. No build or regression test was run because this review invocation is explicitly review-only; current CI shows FE CheckStyle passed.
  • User focus: no additional user-provided review focus was supplied; the whole PR was reviewed.

Review completion: INCOMPLETE at the three-round cap. A distinct table-name-contract finding was identified in Round 3, so the final three-comment payload could not receive another convergence round. All substantiated findings identified within the allowed rounds are included here.

Comment thread regression-test/suites/row_binlog_p0/test_row_binlog_cluster_key.groovy Outdated
Comment thread regression-test/suites/row_binlog_p0/test_row_binlog_cluster_key.groovy Outdated
Comment thread regression-test/suites/row_binlog_p0/test_row_binlog_cluster_key.groovy Outdated
@csun5285
csun5285 force-pushed the forbid-row-binlog-cluster-key branch from 12be97e to efaece7 Compare July 27, 2026 09:05
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: Request changes

Findings

  • Regenerate the SHOW CREATE golden under the standard binlog-enabled regression configuration; the committed row omits the effective binlog fields that CI renders.
  • Pin storage_format in the successful CREATE before recording its full SHOW CREATE output; nonConcurrent fuzzy sessions otherwise choose V2 or V3 randomly while the golden requires V2.

Critical checkpoint conclusions

  • Goal and data correctness: the production change correctly prevents enabled ROW binlog from coexisting with MOW cluster keys. Nereids suppresses testing-only random cluster keys when ROW binlog is effective, and the catalog independently recomputes the effective config and rejects explicit or generated cluster keys before table registration or tablet creation.
  • Scope and clarity: the Java change is small and reuses one locked database-plus-table config snapshot helper rather than duplicating inheritance logic.
  • Concurrency and lifecycle: the database read lock protects only a short config copy; no heavy work, nested lock, escaping shared state, static initialization, or special lifetime is added. Either direction of a database-property change between analysis and execution remains fail-safe at the final catalog guard.
  • Compatibility and parallel paths: direct CREATE, CTAS, CREATE LIKE, and supported programmatic CreateTableCommand users converge on the same final guard. ALTER cannot change an existing binlog format or toggle enable once the format is ROW, and replay preserves historical metadata for compatibility. No storage format, edit-log shape, function symbol, or FE-BE protocol changes are introduced.
  • Error handling, persistence, and data writes: the incompatibility raises through the existing AnalysisException-to-DdlException boundary before persistent registration or external tablet work. No transaction visibility or data-write path changes.
  • Tests: the negative case, inherited database-format/table-enable case, named query cases, pre-test cleanup, and preserved post-test state are structurally appropriate. However, the new SHOW CREATE oracle is deterministically wrong under the standard binlog-enabled FE and independently flaky under fuzzy V3 selection. No build or regression execution was attempted because this review invocation is explicitly review-only.
  • Performance and observability: the added work is a small metadata copy plus a fuzz-only catalog lookup. The explicit error and existing table-properties/SHOW CREATE surfaces provide sufficient diagnostics; no new metric is warranted.
  • User focus: no additional user-provided focus was supplied, so the whole PR was reviewed.

Review coverage is complete after two convergence rounds: every candidate was accepted, duplicated, or dismissed with concrete evidence, and all normal and risk-focused reviewers returned NO_NEW_VALUABLE_FINDINGS for this exact final payload.

Comment thread regression-test/data/row_binlog_p0/test_row_binlog_cluster_key.out Outdated
@csun5285
csun5285 force-pushed the forbid-row-binlog-cluster-key branch from efaece7 to c70a537 Compare July 27, 2026 10:46
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 2nd, 2026 1:27 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30259452831

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29234 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c70a537544ead48d9d645084862d00680349c0eb, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17588	4138	4095	4095
q2	2029	316	205	205
q3	10319	1410	789	789
q4	4705	470	337	337
q5	7587	839	559	559
q6	214	178	138	138
q7	781	821	622	622
q8	9911	1518	1581	1518
q9	6114	4331	4388	4331
q10	6796	1720	1485	1485
q11	521	344	331	331
q12	742	571	457	457
q13	18148	3438	2745	2745
q14	281	271	251	251
q15	q16	791	782	710	710
q17	2611	1201	739	739
q18	7072	5747	5507	5507
q19	1926	1232	1040	1040
q20	797	698	609	609
q21	6022	2576	2466	2466
q22	442	359	300	300
Total cold run time: 105397 ms
Total hot run time: 29234 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4495	4351	4365	4351
q2	290	311	216	216
q3	4582	4943	4471	4471
q4	2050	2217	1371	1371
q5	4426	4258	4395	4258
q6	315	223	153	153
q7	2141	1822	1635	1635
q8	2567	2262	2193	2193
q9	8226	7884	7721	7721
q10	4673	4668	4290	4290
q11	598	443	439	439
q12	769	766	558	558
q13	3279	3561	2957	2957
q14	297	314	279	279
q15	q16	704	730	650	650
q17	1402	1494	1366	1366
q18	7959	7452	7254	7254
q19	1106	1080	1100	1080
q20	2211	2213	1963	1963
q21	5289	4598	4510	4510
q22	511	458	410	410
Total cold run time: 57890 ms
Total hot run time: 52125 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 178934 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c70a537544ead48d9d645084862d00680349c0eb, data reload: false

query5	4334	644	498	498
query6	460	224	213	213
query7	4861	604	349	349
query8	371	187	173	173
query9	8772	4077	4056	4056
query10	474	361	304	304
query11	5959	2319	2129	2129
query12	157	104	98	98
query13	1277	609	414	414
query14	6226	5224	4912	4912
query14_1	4287	4259	4234	4234
query15	232	208	184	184
query16	1045	504	470	470
query17	1131	723	591	591
query18	2438	485	359	359
query19	217	200	155	155
query20	120	110	108	108
query21	235	161	136	136
query22	13571	13683	13334	13334
query23	17410	16437	16078	16078
query23_1	16322	16296	16195	16195
query24	7611	1700	1294	1294
query24_1	1299	1291	1297	1291
query25	566	473	401	401
query26	1347	346	223	223
query27	2593	634	389	389
query28	4499	1990	2014	1990
query29	1107	652	507	507
query30	354	264	233	233
query31	1124	1091	1006	1006
query32	134	69	63	63
query33	550	335	267	267
query34	1175	1141	648	648
query35	791	777	677	677
query36	1183	1168	1079	1079
query37	155	109	94	94
query38	1900	1707	1626	1626
query39	878	870	862	862
query39_1	848	818	837	818
query40	260	175	151	151
query41	72	71	69	69
query42	97	96	94	94
query43	334	332	304	304
query44	1469	786	772	772
query45	206	189	179	179
query46	1045	1207	768	768
query47	2107	2117	2023	2023
query48	415	436	317	317
query49	607	427	325	325
query50	1103	432	345	345
query51	10765	10809	10753	10753
query52	90	96	78	78
query53	266	287	210	210
query54	300	265	229	229
query55	78	76	68	68
query56	340	327	326	326
query57	1336	1333	1172	1172
query58	286	273	259	259
query59	1588	1661	1462	1462
query60	296	280	260	260
query61	156	153	148	148
query62	544	488	428	428
query63	252	214	216	214
query64	2880	1033	856	856
query65	4693	4609	4649	4609
query66	1826	505	413	413
query67	29236	29319	29214	29214
query68	3139	1653	1020	1020
query69	442	307	271	271
query70	1061	945	972	945
query71	382	348	314	314
query72	3026	2675	2434	2434
query73	824	803	410	410
query74	5089	4972	4725	4725
query75	2537	2564	2153	2153
query76	2303	1199	796	796
query77	351	373	272	272
query78	11856	11867	11377	11377
query79	1370	1153	723	723
query80	1288	540	469	469
query81	530	339	286	286
query82	896	156	123	123
query83	392	338	309	309
query84	288	161	134	134
query85	1016	615	529	529
query86	456	298	278	278
query87	1839	1820	1760	1760
query88	3731	2830	2777	2777
query89	465	383	325	325
query90	1919	192	203	192
query91	205	197	168	168
query92	66	61	58	58
query93	1667	1519	956	956
query94	748	355	350	350
query95	830	536	464	464
query96	1055	802	360	360
query97	2636	2598	2503	2503
query98	209	217	201	201
query99	1091	1110	982	982
Total cold run time: 264490 ms
Total hot run time: 178934 ms

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 2nd, 2026 1:27 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30259452831

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c70a537544ead48d9d645084862d00680349c0eb, data reload: false

query1	0.00	0.00	0.01
query2	0.14	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.15	0.14
query5	0.25	0.22	0.23
query6	1.27	1.10	1.11
query7	0.03	0.00	0.00
query8	0.06	0.03	0.03
query9	0.38	0.30	0.33
query10	0.57	0.55	0.55
query11	0.18	0.14	0.13
query12	0.18	0.15	0.14
query13	0.47	0.47	0.49
query14	1.03	1.01	0.99
query15	0.62	0.60	0.60
query16	0.33	0.32	0.31
query17	1.13	1.16	1.13
query18	0.24	0.21	0.22
query19	2.12	2.07	2.08
query20	0.02	0.02	0.01
query21	16.79	0.23	0.14
query22	5.09	0.06	0.05
query23	16.16	0.31	0.11
query24	3.02	0.42	0.32
query25	0.11	0.06	0.04
query26	0.76	0.21	0.15
query27	0.03	0.04	0.04
query28	3.50	0.93	0.54
query29	12.49	4.14	3.29
query30	0.28	0.17	0.16
query31	2.78	0.62	0.31
query32	3.22	0.59	0.50
query33	3.22	3.10	3.32
query34	15.52	4.23	3.56
query35	3.52	3.56	3.56
query36	0.55	0.44	0.41
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.04	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.44 s
Total hot run time: 25.21 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 52.63% (10/19) 🎉
Increment coverage report
Complete coverage report

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 2nd, 2026 1:27 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30320771414

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Your access token could not be refreshed because your refresh token was revoked. Please log out and sign in again.
Workflow run: https://github.com/apache/doris/actions/runs/30322800497

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@shuke987

Copy link
Copy Markdown
Contributor

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: No new blocking code-review issues found on c70a537544ead48d9d645084862d00680349c0eb.

Critical checkpoint conclusions:

  • Goal and test proof: The final catalog guard rejects enabled ROW binlog on MOW tables with cluster keys before tablet creation, while Nereids suppresses the testing-only random cluster-key injection when ROW binlog is effective. The negative regression and inherited-config golden cases cover those behaviors compositionally.
  • Scope, clarity, and reuse: The production change is focused and reuses one database snapshot/effective-config helper in validation and execution.
  • Concurrency and lifecycle: Database config copies are taken under the database read lock; mutations use the write lock. Validation/execution snapshot drift is fail-safe at the execution guard and adds no lock-order or object-lifetime hazard.
  • Configuration and compatibility: No new config, persisted field, storage format, function symbol, or FE-BE protocol value is introduced. The existing mutable fuzzy-test config is scoped and restored. CREATE, CTAS, CREATE TABLE LIKE, supported programmatic CREATE, ALTER, replay, and restore paths were checked.
  • Error handling, persistence, and data writes: The incompatibility raises through the existing analysis-to-DDL failure boundary before row-binlog metadata, tablet work, or table registration. No transaction visibility or data-write path changes.
  • Tests and results: The current suite uses the required expected-error form, pre-test cleanup, named golden queries, deterministic ordering, pinned storage format, and preserved post-test state. The generated golden records the effective ROW properties and absence of random ORDER BY. The five prior inline concerns are addressed on the current head.
  • Performance and observability: Added work is a short metadata copy plus a lookup behind the fuzzy-only branch; no material hot-path cost or new metric is warranted. The specific error plus existing properties/SHOW CREATE surfaces are sufficient.
  • User focus: No additional user-provided focus was supplied; the whole PR was reviewed.
  • Validation limit: This invocation is review-only, so no local build or test was run. Live Compile, CheckStyle, FE UT, P0, NonConcurrent, and performance checks pass. check_coverage_fe remains failed, but its TeamCity log returns HTTP 401 here, so this review does not infer a patch defect from that opaque status.

Review completion: COMPLETE after three rounds. Every candidate was accepted, duplicated, or dismissed with concrete evidence, and all normal plus risk-focused reviewers returned NO_NEW_VALUABLE_FINDINGS for the final resolved ledger and empty inline-comment set.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@csun5285
csun5285 merged commit 7488bc7 into apache:master Jul 28, 2026
30 of 31 checks passed
csun5285 added a commit to csun5285/doris that referenced this pull request Jul 28, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66093

Problem Summary: PR apache#66093 rejects Row Binlog on unique MOW tables with cluster keys during table creation. Remove the now-unreachable Segment-format subcase and its two golden files, and restore the historical-row lookup call that was changed only to support that combination. Keep the independent MOW cluster-key cases and all supported Row Binlog, sequence, BEFORE, and compaction coverage.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `./run-be-ut.sh --run --filter=<all-six-segment-flusher-golden-producers> -j 32` (6 tests passed)
    - `build-support/check-format.sh` passed with clang-format 16.0.6
    - clang-tidy reported no warning for the changed test file; analysis of `base_tablet.cpp` is blocked by the pre-existing unmatched `NOLINTEND` in `be/src/core/types.h:576`
- Behavior changed: No
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants