Skip to content

[fix](storage) Reject prepare txn on shutdown tablet - #66448

Merged
deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-prepare-txn-shutdown-tablet
Aug 5, 2026
Merged

[fix](storage) Reject prepare txn on shutdown tablet#66448
deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-prepare-txn-shutdown-tablet

Conversation

@deardeng

@deardeng deardeng commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Cause:
PR #54124 moved the migration and push locks from RowsetBuilder into Tablet::prepare_txn. During that refactor, the call to TxnManager::prepare_txn changed from the const Tablet& overload to the raw tablet_id/tablet_uid overload. This silently bypassed the TABLET_SHUTDOWN guard introduced by PR #42296.

Impact:
A load thread holding an old Tablet instance could resume after migration and register the stale tablet UID in the transaction map. Publish would then operate on the reloaded Tablet with a new UID and leave the transaction in an E-909 state.

Fix:
Call the const Tablet& overload with *this while the migration and push locks are held. This preserves the lock encapsulation from PR #54124 and restores the shutdown-tablet protection from PR #42296.

Test:
Add a deterministic unit test that marks a Tablet as TABLET_SHUTDOWN, verifies Tablet::prepare_txn fails, and verifies no stale tablet entry is registered. TxnManagerTest.* passes 16/16.

(cherry picked from commit ccfb166677719b65dd224970e0f2323a1e93c1fb)

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Cause:
PR apache#54124 moved the migration and push locks from RowsetBuilder into
Tablet::prepare_txn. During that refactor, the call to
TxnManager::prepare_txn changed from the const Tablet& overload to the raw
tablet_id/tablet_uid overload. This silently bypassed the TABLET_SHUTDOWN
guard introduced by PR apache#42296.

Impact:
A load thread holding an old Tablet instance could resume after migration
and register the stale tablet UID in the transaction map. Publish would then
operate on the reloaded Tablet with a new UID and leave the transaction in an
E-909 state.

Fix:
Call the const Tablet& overload with *this while the migration and push locks
are held. This preserves the lock encapsulation from PR apache#54124 and restores
the shutdown-tablet protection from PR apache#42296.

Test:
Add a deterministic unit test that marks a Tablet as TABLET_SHUTDOWN, verifies
Tablet::prepare_txn fails, and verifies no stale tablet entry is registered.
TxnManagerTest.* passes 16/16.

(cherry picked from commit ccfb166677719b65dd224970e0f2323a1e93c1fb)
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng

deardeng commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@deardeng

deardeng commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28715 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01e76ad40db17790e3e4381046f618834ba6e61a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17732	3954	3890	3890
q2	2019	321	197	197
q3	10302	1373	785	785
q4	4677	468	335	335
q5	7532	843	548	548
q6	177	167	136	136
q7	715	803	600	600
q8	9567	1517	1569	1517
q9	5750	4075	4014	4014
q10	6818	1604	1344	1344
q11	526	351	314	314
q12	723	579	436	436
q13	18076	3164	2754	2754
q14	261	253	240	240
q15	q16	725	727	656	656
q17	1008	962	957	957
q18	6488	5600	5548	5548
q19	1173	1291	1115	1115
q20	792	696	615	615
q21	5660	2576	2415	2415
q22	421	360	299	299
Total cold run time: 101142 ms
Total hot run time: 28715 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4224	4115	4131	4115
q2	281	314	216	216
q3	4514	4895	4312	4312
q4	2161	2221	1391	1391
q5	4197	4066	4073	4066
q6	224	168	124	124
q7	1679	1599	1367	1367
q8	2313	2267	2063	2063
q9	7247	7164	7324	7164
q10	4291	4246	3851	3851
q11	551	404	369	369
q12	693	713	499	499
q13	3162	3450	3087	3087
q14	309	298	280	280
q15	q16	713	716	620	620
q17	1258	1257	1281	1257
q18	12130	11058	11735	11058
q19	1136	1123	1110	1110
q20	2225	2226	1936	1936
q21	5566	4832	4823	4823
q22	535	473	418	418
Total cold run time: 59409 ms
Total hot run time: 54126 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 165875 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01e76ad40db17790e3e4381046f618834ba6e61a, data reload: false

query5	4317	591	445	445
query6	471	210	198	198
query7	4859	587	337	337
query8	314	157	144	144
query9	8740	3982	4008	3982
query10	484	360	297	297
query11	5808	2185	1989	1989
query12	152	100	93	93
query13	1278	595	434	434
query14	6031	4261	3986	3986
query14_1	3773	3762	3787	3762
query15	207	205	175	175
query16	1019	481	490	481
query17	925	708	524	524
query18	2414	487	335	335
query19	206	184	139	139
query20	99	96	99	96
query21	227	154	139	139
query22	13118	12995	12763	12763
query23	15652	15059	14487	14487
query23_1	14636	14577	14644	14577
query24	7593	1709	1224	1224
query24_1	1266	1234	1223	1223
query25	534	424	348	348
query26	1295	368	213	213
query27	2615	589	382	382
query28	4560	2082	2066	2066
query29	1075	600	462	462
query30	341	257	220	220
query31	1191	1103	1031	1031
query32	113	65	63	63
query33	529	323	266	266
query34	1189	1133	658	658
query35	740	747	655	655
query36	798	771	713	713
query37	160	109	96	96
query38	1827	1769	1716	1716
query39	830	828	817	817
query39_1	783	788	786	786
query40	263	172	147	147
query41	70	75	69	69
query42	98	93	95	93
query43	312	332	275	275
query44	1420	794	775	775
query45	187	180	177	177
query46	1022	1185	714	714
query47	1567	1532	1494	1494
query48	407	430	309	309
query49	590	437	294	294
query50	1066	429	337	337
query51	10634	10364	10604	10364
query52	81	85	71	71
query53	253	268	200	200
query54	280	232	213	213
query55	73	69	65	65
query56	308	307	283	283
query57	1028	1009	918	918
query58	292	271	236	236
query59	1533	1619	1417	1417
query60	323	274	254	254
query61	148	145	152	145
query62	394	318	266	266
query63	241	189	191	189
query64	2893	1027	886	886
query65	3893	3827	3810	3810
query66	1833	479	349	349
query67	27475	28055	28064	28055
query68	3129	1601	1000	1000
query69	402	305	260	260
query70	864	772	773	772
query71	374	337	301	301
query72	2977	2617	2302	2302
query73	789	794	425	425
query74	4616	4462	4308	4308
query75	2368	2323	1983	1983
query76	2322	1118	730	730
query77	349	362	275	275
query78	11074	10968	10535	10535
query79	1290	1136	776	776
query80	644	570	456	456
query81	452	319	282	282
query82	489	172	134	134
query83	403	316	312	312
query84	334	157	129	129
query85	934	610	531	531
query86	316	232	219	219
query87	1959	1978	1822	1822
query88	3713	2800	2729	2729
query89	381	316	284	284
query90	1860	200	190	190
query91	197	191	163	163
query92	64	59	54	54
query93	1565	1539	987	987
query94	536	361	314	314
query95	803	600	502	502
query96	1053	755	358	358
query97	2498	2458	2374	2374
query98	198	190	186	186
query99	759	744	624	624
Total cold run time: 250789 ms
Total hot run time: 165875 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.98 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01e76ad40db17790e3e4381046f618834ba6e61a, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.06	0.05
query3	0.26	0.14	0.14
query4	1.60	0.14	0.14
query5	0.23	0.22	0.22
query6	1.16	0.83	0.81
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.38	0.32	0.32
query10	0.55	0.54	0.58
query11	0.18	0.13	0.14
query12	0.18	0.14	0.14
query13	0.46	0.47	0.46
query14	1.01	0.99	0.98
query15	0.62	0.60	0.62
query16	0.32	0.31	0.33
query17	1.07	1.10	1.14
query18	0.21	0.20	0.20
query19	2.06	1.97	1.97
query20	0.02	0.01	0.02
query21	15.44	0.22	0.13
query22	4.82	0.05	0.05
query23	16.13	0.31	0.13
query24	2.88	0.44	0.33
query25	0.10	0.05	0.05
query26	0.72	0.22	0.16
query27	0.05	0.03	0.04
query28	3.49	0.76	0.36
query29	12.48	4.03	3.16
query30	0.27	0.15	0.15
query31	2.78	0.57	0.32
query32	3.23	0.59	0.49
query33	3.12	3.16	3.23
query34	15.71	3.93	3.26
query35	3.25	3.23	3.22
query36	0.55	0.46	0.41
query37	0.09	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.09	0.04	0.04
query42	0.03	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.06 s
Total hot run time: 23.98 s

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review result: request changes

The overload switch correctly restores the shutdown guard for ordinary Tablet::prepare_txn callers, but two P1 lifecycle gaps can still record a retired tablet UID and fail exact-UID publish.

Critical checkpoint conclusions:

  • Goal and correctness: the changed path fixes the sequential overload-dispatch regression, but the end-to-end stale-tablet goal remains incomplete because the slave direct-commit path bypasses prepare and retained tablets are not fenced through commit against other shutdown/replacement paths.
  • Scope and parallel paths: the patch is minimal and focused; RowsetBuilder, realtime push, and backend-binlog ingestion converge on the guarded path and propagate its Status, while the slave pull is the relevant live bypass.
  • Concurrency and lifecycle: the intended disk-migration lock ordering is consistent and no new deadlock was found. Bad-tablet repair, manual reload/replacement, and one-shot FE drop cleanup do not provide an equivalent prepare-to-commit/current-object fence.
  • Transactions, persistence, and data correctness: the remaining interleavings can persist or insert an old (tablet_id, tablet_uid) identity; publish later requires that exact identity and cannot resolve the replacement.
  • Tests: the added unit test deterministically proves overload dispatch and absence of a prepared map entry for an already-shutdown tablet. It does not cover slave-pull/migration or retained-pointer shutdown/prepare/commit interleavings. No local builds or tests were run because the authoritative review prompt requires static-only review.
  • Error handling and observability: ordinary callers already propagate the failure, and the guarded error identifies the tablet; no additional observability issue was found.
  • Configuration, compatibility, ownership, memory, and performance: no applicable regression was found; the restored state check is constant-time.
  • Review coverage: no additional user focus was supplied, the full two-file diff was reviewed, and no existing inline thread duplicates either finding.
  • Completion: three review rounds were used; all three final-round reviewers returned NO_NEW_VALUABLE_FINDINGS against these exact comments and diff position 6.

Comment thread be/src/storage/tablet/tablet.cpp
Comment thread be/src/storage/tablet/tablet.cpp
@deardeng deardeng removed the dev/3.1.x label Aug 4, 2026
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

Copy link
Copy Markdown
Contributor

skip check_coverage

@deardeng
deardeng merged commit 5a24e63 into apache:master Aug 5, 2026
35 of 36 checks passed
github-actions Bot pushed a commit that referenced this pull request Aug 5, 2026
Cause:
PR #54124 moved the migration and push locks from RowsetBuilder into
Tablet::prepare_txn. During that refactor, the call to
TxnManager::prepare_txn changed from the const Tablet& overload to the
raw tablet_id/tablet_uid overload. This silently bypassed the
TABLET_SHUTDOWN guard introduced by PR #42296.

Impact:
A load thread holding an old Tablet instance could resume after
migration and register the stale tablet UID in the transaction map.
Publish would then operate on the reloaded Tablet with a new UID and
leave the transaction in an E-909 state.

Fix:
Call the const Tablet& overload with *this while the migration and push
locks are held. This preserves the lock encapsulation from PR #54124 and
restores the shutdown-tablet protection from PR #42296.

Test:
Add a deterministic unit test that marks a Tablet as TABLET_SHUTDOWN,
verifies Tablet::prepare_txn fails, and verifies no stale tablet entry
is registered. TxnManagerTest.* passes 16/16.
github-actions Bot pushed a commit that referenced this pull request Aug 5, 2026
Cause:
PR #54124 moved the migration and push locks from RowsetBuilder into
Tablet::prepare_txn. During that refactor, the call to
TxnManager::prepare_txn changed from the const Tablet& overload to the
raw tablet_id/tablet_uid overload. This silently bypassed the
TABLET_SHUTDOWN guard introduced by PR #42296.

Impact:
A load thread holding an old Tablet instance could resume after
migration and register the stale tablet UID in the transaction map.
Publish would then operate on the reloaded Tablet with a new UID and
leave the transaction in an E-909 state.

Fix:
Call the const Tablet& overload with *this while the migration and push
locks are held. This preserves the lock encapsulation from PR #54124 and
restores the shutdown-tablet protection from PR #42296.

Test:
Add a deterministic unit test that marks a Tablet as TABLET_SHUTDOWN,
verifies Tablet::prepare_txn fails, and verifies no stale tablet entry
is registered. TxnManagerTest.* passes 16/16.
yiguolei pushed a commit that referenced this pull request Aug 6, 2026
… (#66457)

Cherry-picked from #66448

Co-authored-by: deardeng <dengxin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.x dev/4.1.4-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants