Skip to content

[fix](load) Fix potential data loss during disk migration #42296 #42385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Oct 24, 2024
Merged

[fix](load) Fix potential data loss during disk migration #42296 #42385

liaoxin01 merged 1 commit into from
Oct 24, 2024

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Oct 24, 2024

cherry pick from #42296

@liaoxin01
[fix](load) Fix potential data loss during disk migration (apache#42296)
ea8b002 
The following operations may trigger this issue.
1. migration start
2. load start using old tablet，but wait migration lock in
`RowsetBuilder::prepare_txn`.
3. migration finish, old tablet -> new tablet
4. obtained migration lock and commit successfully using old tablet.
5. publish failed using old tablet, because old tablet has been dropped.
It cause the data loss.

Therefore, after acquiring the migration lock, check if the tablet has
already been shut down. If it has, it indicates that it is an old
tablet, and data should not be imported into the old tablet.
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 24, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

TPC-H: Total hot run time: 40580 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea8b0022858ca8bd6ee9a846ee5b97a347c4ae43, data reload: false

------ Round 1 ----------------------------------
q1	17581	7368	7279	7279
q2	2006	281	271	271
q3	12227	1052	1166	1052
q4	10538	743	774	743
q5	7740	2880	2819	2819
q6	237	156	147	147
q7	989	630	595	595
q8	9557	1895	1968	1895
q9	7992	6416	6413	6413
q10	6985	2277	2328	2277
q11	430	249	259	249
q12	451	222	216	216
q13	17760	2948	2955	2948
q14	232	209	210	209
q15	564	525	515	515
q16	675	623	613	613
q17	967	564	564	564
q18	7112	6632	6557	6557
q19	1388	954	1078	954
q20	487	201	207	201
q21	3896	3105	3205	3105
q22	1115	958	992	958
Total cold run time: 110929 ms
Total hot run time: 40580 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7221	7262	7198	7198
q2	322	229	229	229
q3	3068	2841	2820	2820
q4	2053	1760	1765	1760
q5	5661	5662	5745	5662
q6	230	142	146	142
q7	2166	1786	1739	1739
q8	3302	3498	3417	3417
q9	8824	8821	8750	8750
q10	3533	3513	3463	3463
q11	574	495	499	495
q12	809	647	580	580
q13	16501	3124	3141	3124
q14	327	288	266	266
q15	571	516	509	509
q16	708	671	670	670
q17	1840	1595	1576	1576
q18	8190	7715	7398	7398
q19	3328	1443	1504	1443
q20	2118	1888	1878	1878
q21	5283	5212	5308	5212
q22	1105	1002	1016	1002
Total cold run time: 77734 ms
Total hot run time: 59333 ms

@doris-robot
Copy link

doris-robot commented Oct 24, 2024

TPC-DS: Total hot run time: 189851 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea8b0022858ca8bd6ee9a846ee5b97a347c4ae43, data reload: false

query1	920	402	385	385
query2	6274	2124	1999	1999
query3	8696	196	200	196
query4	33595	23404	23565	23404
query5	3722	467	447	447
query6	292	170	169	169
query7	4202	322	310	310
query8	276	211	215	211
query9	9467	2659	2617	2617
query10	473	281	284	281
query11	17801	15133	15308	15133
query12	169	96	94	94
query13	1518	412	399	399
query14	9509	7154	7178	7154
query15	244	166	170	166
query16	7678	456	437	437
query17	1614	584	559	559
query18	1400	323	303	303
query19	222	155	162	155
query20	124	121	111	111
query21	216	109	117	109
query22	4551	4354	4241	4241
query23	34780	34001	33886	33886
query24	11031	2881	2859	2859
query25	621	407	397	397
query26	727	167	162	162
query27	2117	306	299	299
query28	6809	2492	2472	2472
query29	698	443	436	436
query30	270	151	159	151
query31	1046	807	767	767
query32	105	53	56	53
query33	770	294	312	294
query34	928	522	505	505
query35	849	751	733	733
query36	1119	953	951	951
query37	139	85	86	85
query38	3952	3869	3807	3807
query39	1491	1437	1414	1414
query40	197	100	95	95
query41	49	49	48	48
query42	129	101	94	94
query43	525	487	488	487
query44	1258	777	790	777
query45	199	165	164	164
query46	1154	706	707	706
query47	1907	1775	1842	1775
query48	469	364	378	364
query49	911	407	387	387
query50	848	410	404	404
query51	7080	6905	6751	6751
query52	99	96	90	90
query53	260	192	191	191
query54	1219	476	466	466
query55	78	80	76	76
query56	296	248	251	248
query57	1186	1108	1101	1101
query58	259	224	225	224
query59	3075	2895	2990	2895
query60	290	268	284	268
query61	108	124	110	110
query62	876	655	662	655
query63	213	181	183	181
query64	4158	631	612	612
query65	3243	3140	3162	3140
query66	844	313	309	309
query67	15821	15378	15153	15153
query68	4470	548	562	548
query69	558	322	283	283
query70	1197	1099	1140	1099
query71	404	279	275	275
query72	7775	3896	3884	3884
query73	770	338	343	338
query74	10249	8863	8863	8863
query75	3917	2651	2657	2651
query76	3874	915	936	915
query77	622	282	269	269
query78	10571	9471	9460	9460
query79	5209	573	582	573
query80	3164	436	448	436
query81	587	250	247	247
query82	1039	142	142	142
query83	304	136	132	132
query84	298	82	75	75
query85	2239	296	281	281
query86	466	286	308	286
query87	4352	4151	4221	4151
query88	4387	2376	2400	2376
query89	413	293	284	284
query90	2076	188	185	185
query91	173	141	163	141
query92	63	48	49	48
query93	4222	550	534	534
query94	1081	301	304	301
query95	356	256	250	250
query96	625	278	286	278
query97	3287	3210	3134	3134
query98	217	198	192	192
query99	1565	1303	1298	1298
Total cold run time: 307696 ms
Total hot run time: 189851 ms

@liaoxin01 liaoxin01 merged commit 725e70d into apache:branch-3.0 Oct 24, 2024
@liaoxin01 liaoxin01 deleted the pick_42296_to_origin_branch-3.0 branch October 24, 2024 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@liaoxin01 @doris-robot