Skip to content

[fix](restore) Add a local snapshot lock to protect snapshot dir#47279

Merged
gavinchou merged 1 commit intoapache:masterfrom
w41ter:local_snapshot_lock
Jan 21, 2025
Merged

[fix](restore) Add a local snapshot lock to protect snapshot dir#47279
gavinchou merged 1 commit intoapache:masterfrom
w41ter:local_snapshot_lock

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Jan 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

To avoid concurrent modification of a snapshot dir.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Jan 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

TPC-H: Total hot run time: 32432 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c6b773a3ff6fcf6a28cf15cb6d0525bf81d547b, data reload: false

------ Round 1 ----------------------------------
q1	17591	5563	5471	5471
q2	2070	331	185	185
q3	10950	1255	744	744
q4	10263	978	537	537
q5	8655	2406	2156	2156
q6	198	163	133	133
q7	897	758	598	598
q8	9238	1386	1165	1165
q9	5183	4884	4891	4884
q10	6864	2359	1899	1899
q11	476	286	264	264
q12	347	365	219	219
q13	17781	3720	3121	3121
q14	235	237	204	204
q15	523	481	473	473
q16	626	620	591	591
q17	577	858	332	332
q18	6890	6436	6369	6369
q19	1883	940	543	543
q20	334	329	188	188
q21	2860	2369	2039	2039
q22	378	335	317	317
Total cold run time: 104819 ms
Total hot run time: 32432 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5563	5492	5494	5492
q2	253	329	239	239
q3	2286	2662	2272	2272
q4	1422	1824	1373	1373
q5	4325	4776	4729	4729
q6	177	161	129	129
q7	2075	1921	1819	1819
q8	2650	2873	2725	2725
q9	7319	7202	7208	7202
q10	3035	3296	2812	2812
q11	559	508	496	496
q12	642	752	596	596
q13	3660	4047	3378	3378
q14	301	296	276	276
q15	521	467	473	467
q16	656	695	654	654
q17	1259	1742	1283	1283
q18	7655	7489	7425	7425
q19	832	1181	1108	1108
q20	2092	2046	1951	1951
q21	5662	5086	5098	5086
q22	596	652	558	558
Total cold run time: 53540 ms
Total hot run time: 52070 ms

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

TeamCity be ut coverage result:
Function Coverage: 41.56% (10834/26068)
Line Coverage: 31.96% (91567/286518)
Region Coverage: 31.10% (46909/150838)
Branch Coverage: 27.18% (23750/87386)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1c6b773a3ff6fcf6a28cf15cb6d0525bf81d547b_1c6b773a3ff6fcf6a28cf15cb6d0525bf81d547b/report/index.html

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

TPC-DS: Total hot run time: 195429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c6b773a3ff6fcf6a28cf15cb6d0525bf81d547b, data reload: false

query1	1364	986	910	910
query2	6352	2135	2090	2090
query3	10958	4451	4543	4451
query4	60665	29198	23477	23477
query5	5526	601	486	486
query6	436	204	189	189
query7	5571	502	293	293
query8	320	227	223	223
query9	8372	2619	2573	2573
query10	470	304	259	259
query11	17454	15040	15792	15040
query12	172	114	104	104
query13	1446	535	396	396
query14	10984	7478	7919	7478
query15	211	198	191	191
query16	6585	684	471	471
query17	1119	757	597	597
query18	1044	383	339	339
query19	203	185	159	159
query20	118	118	121	118
query21	219	122	106	106
query22	4464	4572	4507	4507
query23	34416	33200	33286	33200
query24	5810	2333	2356	2333
query25	499	480	426	426
query26	648	285	157	157
query27	1684	492	353	353
query28	3797	2483	2466	2466
query29	559	577	437	437
query30	215	188	166	166
query31	924	887	815	815
query32	79	61	59	59
query33	451	355	311	311
query34	740	895	508	508
query35	820	846	743	743
query36	995	1059	974	974
query37	129	103	79	79
query38	4293	4409	4247	4247
query39	1493	1486	1631	1486
query40	215	117	119	117
query41	52	52	55	52
query42	123	108	107	107
query43	531	540	490	490
query44	1421	845	852	845
query45	184	191	175	175
query46	899	1065	662	662
query47	1903	1882	1846	1846
query48	399	423	336	336
query49	731	499	405	405
query50	674	683	417	417
query51	7020	7024	7050	7024
query52	115	103	92	92
query53	245	256	189	189
query54	506	516	454	454
query55	102	81	81	81
query56	263	282	269	269
query57	1209	1184	1103	1103
query58	247	224	244	224
query59	3155	3335	3223	3223
query60	273	270	252	252
query61	113	114	119	114
query62	747	703	652	652
query63	220	199	187	187
query64	1309	1066	662	662
query65	3230	3199	3141	3141
query66	711	399	300	300
query67	16134	15716	15292	15292
query68	5032	843	520	520
query69	484	294	258	258
query70	1208	1130	1152	1130
query71	399	291	268	268
query72	6070	3925	3808	3808
query73	783	756	359	359
query74	10261	8934	8807	8807
query75	3277	3153	2704	2704
query76	3802	1185	768	768
query77	477	372	285	285
query78	10075	10032	9476	9476
query79	3132	809	598	598
query80	685	528	471	471
query81	504	274	246	246
query82	572	156	120	120
query83	180	171	151	151
query84	284	101	72	72
query85	753	344	293	293
query86	380	312	287	287
query87	4419	4563	4432	4432
query88	4719	2155	2140	2140
query89	401	324	312	312
query90	1637	190	195	190
query91	135	142	107	107
query92	68	58	52	52
query93	2836	877	529	529
query94	738	410	304	304
query95	330	272	256	256
query96	521	615	335	335
query97	2848	2839	2744	2744
query98	230	203	192	192
query99	1289	1364	1287	1287
Total cold run time: 312865 ms
Total hot run time: 195429 ms

@doris-robot
Copy link

doris-robot commented Jan 21, 2025

ClickBench: Total hot run time: 30.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c6b773a3ff6fcf6a28cf15cb6d0525bf81d547b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.41	0.42	0.41
query6	1.18	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.61	0.50	0.48
query10	0.56	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.61
query14	2.71	2.73	2.76
query15	0.88	0.83	0.83
query16	0.39	0.39	0.38
query17	1.06	1.05	1.05
query18	0.24	0.21	0.20
query19	1.91	1.83	1.92
query20	0.02	0.01	0.01
query21	15.36	0.94	0.56
query22	0.76	0.82	0.62
query23	15.31	1.49	0.59
query24	2.61	1.41	0.97
query25	0.16	0.19	0.10
query26	0.30	0.16	0.16
query27	0.05	0.05	0.05
query28	14.06	1.03	0.43
query29	12.54	3.94	3.24
query30	0.25	0.09	0.06
query31	2.83	0.61	0.38
query32	3.22	0.56	0.47
query33	2.98	3.12	3.03
query34	16.56	5.12	4.46
query35	4.52	4.50	4.47
query36	0.64	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.57 s
Total hot run time: 30.59 s

dataroaring
dataroaring approved these changes Jan 21, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit bac5807 into apache:master Jan 21, 2025
26 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 21, 2025
@w41ter
[fix](restore) Add a local snapshot lock to protect snapshot dir (#47279
62ce0fe 
)

To avoid concurrent modification of a snapshot dir.
@github-actions github-actions bot mentioned this pull request Jan 21, 2025
Merged
@w41ter w41ter deleted the local_snapshot_lock branch January 22, 2025 02:14
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Feb 7, 2025
@w41ter
[fix](restore) Add a local snapshot lock to protect snapshot dir (apa…
a23174d 
…che#47279)

To avoid concurrent modification of a snapshot dir.
@w41ter w41ter mentioned this pull request Feb 7, 2025
Merged
w41ter added a commit that referenced this pull request Feb 7, 2025
@w41ter
[fix](restore) Add a local snapshot lock to protect snapshot dir (#47279
2168a44 
)

To avoid concurrent modification of a snapshot dir.
yiguolei pushed a commit that referenced this pull request Feb 8, 2025
@w41ter
branch-2.1: [fix](restore) Add a local snapshot lock to protect snaps…
9d3ee27 
…hot dir #47279 (#47574)

cherry pick from #47279
dataroaring pushed a commit that referenced this pull request Feb 19, 2025
@github-actions @w41ter
branch-3.0: [fix](restore) Add a local snapshot lock to protect snaps…
551dbbb 
…hot dir #47279 (#47292)

Cherry-picked from #47279

Co-authored-by: walter <maochuan@selectdb.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@w41ter @lzyy2024
[fix](restore) Add a local snapshot lock to protect snapshot dir (apa…
5bf55b5 
…che#47279)

To avoid concurrent modification of a snapshot dir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter @hello-stephen @doris-robot @gavinchou @dataroaring @yiguolei