[fix](backup) make snapshot task with unique signature by w41ter · Pull Request #48238 · apache/doris

w41ter · 2025-02-24T08:43:49Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

PR #47416 already filters the staled responses by the job ID. But the task is getting by (backendId, taskType, signature), so some staled responses might not be able to be filtered, if the two signatures are equal.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

w41ter · 2025-02-24T08:43:59Z

run buildall

Thearas · 2025-02-24T08:43:59Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

doris-robot · 2025-02-24T09:08:19Z

TPC-H: Total hot run time: 31452 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9deac0fdd4f4053ca8ce36fb6689613aa17d439, data reload: false

------ Round 1 ----------------------------------
q1	17585	5295	5103	5103
q2	2047	287	178	178
q3	10421	1306	689	689
q4	10218	1012	515	515
q5	7517	2461	2361	2361
q6	192	167	136	136
q7	911	758	616	616
q8	9317	1339	1148	1148
q9	4902	4615	4643	4615
q10	6797	2287	1905	1905
q11	469	269	249	249
q12	344	351	219	219
q13	17768	3659	3082	3082
q14	230	234	207	207
q15	501	468	455	455
q16	617	609	571	571
q17	571	864	352	352
q18	7034	6149	6167	6149
q19	1211	933	525	525
q20	321	323	190	190
q21	2742	2156	1898	1898
q22	367	323	289	289
Total cold run time: 102082 ms
Total hot run time: 31452 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5081	5060	5048	5048
q2	238	336	231	231
q3	2134	2688	2279	2279
q4	1382	1828	1363	1363
q5	4222	4149	4142	4142
q6	203	162	123	123
q7	1866	1804	1655	1655
q8	2583	2689	2518	2518
q9	7393	7087	7046	7046
q10	2947	3188	2785	2785
q11	564	507	489	489
q12	702	743	617	617
q13	3485	3933	3243	3243
q14	265	289	275	275
q15	508	455	461	455
q16	650	680	626	626
q17	1117	1583	1327	1327
q18	7511	7355	7385	7355
q19	773	798	967	798
q20	2000	2009	1905	1905
q21	5366	4990	4863	4863
q22	636	555	544	544
Total cold run time: 51626 ms
Total hot run time: 49687 ms

doris-robot · 2025-02-24T09:19:44Z

TPC-DS: Total hot run time: 183817 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9deac0fdd4f4053ca8ce36fb6689613aa17d439, data reload: false

query1	969	374	375	374
query2	6524	1850	1829	1829
query3	6804	216	208	208
query4	26216	23653	23379	23379
query5	4338	672	509	509
query6	284	188	175	175
query7	4601	487	294	294
query8	298	244	232	232
query9	8632	2555	2556	2555
query10	487	315	249	249
query11	15697	15223	15174	15174
query12	167	111	109	109
query13	1671	508	408	408
query14	9677	6127	6545	6127
query15	221	195	179	179
query16	7530	611	438	438
query17	1191	710	556	556
query18	1980	405	318	318
query19	212	196	170	170
query20	122	121	116	116
query21	215	133	105	105
query22	4408	4493	4638	4493
query23	34521	33276	32893	32893
query24	8019	2353	2362	2353
query25	523	470	387	387
query26	1225	278	150	150
query27	2624	483	327	327
query28	4310	2404	2381	2381
query29	701	530	411	411
query30	229	194	153	153
query31	940	870	740	740
query32	71	63	60	60
query33	564	360	319	319
query34	762	853	512	512
query35	781	818	741	741
query36	964	1004	883	883
query37	121	97	76	76
query38	4058	4151	3998	3998
query39	1450	1377	1358	1358
query40	212	113	101	101
query41	56	50	51	50
query42	125	103	99	99
query43	499	497	446	446
query44	1238	773	766	766
query45	179	175	157	157
query46	853	1059	645	645
query47	1743	1773	1738	1738
query48	393	421	301	301
query49	776	476	421	421
query50	675	728	431	431
query51	4212	4204	4126	4126
query52	109	105	94	94
query53	221	256	184	184
query54	477	484	403	403
query55	79	79	83	79
query56	253	279	260	260
query57	1132	1161	1050	1050
query58	239	246	234	234
query59	2549	2548	2540	2540
query60	308	271	270	270
query61	119	118	113	113
query62	783	703	674	674
query63	221	191	192	191
query64	4336	973	660	660
query65	3216	3097	3126	3097
query66	1082	406	298	298
query67	15981	15740	15660	15660
query68	6766	769	514	514
query69	463	285	255	255
query70	1193	1138	1070	1070
query71	421	291	262	262
query72	5617	3581	3702	3581
query73	712	753	368	368
query74	9205	9121	8951	8951
query75	3140	3190	2674	2674
query76	3240	1157	732	732
query77	461	366	275	275
query78	10019	10119	9261	9261
query79	1945	888	600	600
query80	750	521	440	440
query81	525	270	231	231
query82	378	124	93	93
query83	178	172	145	145
query84	247	91	69	69
query85	762	347	358	347
query86	353	299	306	299
query87	4427	4423	4396	4396
query88	2921	2204	2213	2204
query89	399	329	286	286
query90	1697	197	194	194
query91	146	135	104	104
query92	63	58	56	56
query93	1181	1041	579	579
query94	667	417	299	299
query95	354	277	254	254
query96	516	525	278	278
query97	2794	2914	2690	2690
query98	239	205	200	200
query99	1295	1390	1283	1283
Total cold run time: 269247 ms
Total hot run time: 183817 ms

doris-robot · 2025-02-24T09:25:03Z

ClickBench: Total hot run time: 30.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9deac0fdd4f4053ca8ce36fb6689613aa17d439, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.64	0.10	0.10
query5	0.41	0.42	0.40
query6	1.15	0.66	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.60	0.52	0.52
query10	0.59	0.57	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.70	2.70	2.72
query15	0.92	0.85	0.85
query16	0.40	0.38	0.37
query17	1.04	1.02	1.03
query18	0.21	0.19	0.19
query19	1.92	1.97	1.79
query20	0.01	0.01	0.01
query21	15.36	0.90	0.54
query22	0.76	1.23	0.65
query23	14.94	1.36	0.62
query24	7.00	2.02	0.85
query25	0.54	0.23	0.16
query26	0.54	0.16	0.14
query27	0.05	0.05	0.04
query28	10.32	0.79	0.43
query29	12.53	3.97	3.26
query30	0.25	0.09	0.06
query31	2.84	0.56	0.38
query32	3.23	0.54	0.46
query33	2.93	3.00	3.00
query34	15.89	5.10	4.47
query35	4.53	4.48	4.54
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.18	0.14	0.14
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 105.76 s
Total hot run time: 30.48 s

to filter the staled responses

w41ter · 2025-02-24T10:44:15Z

run buildall

doris-robot · 2025-02-24T11:28:46Z

TPC-H: Total hot run time: 31613 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc1301627bd1cfc23e1534519e1eef3b0f753f10, data reload: false

------ Round 1 ----------------------------------
q1	17616	5577	5124	5124
q2	2054	308	167	167
q3	10429	1233	784	784
q4	10211	1049	543	543
q5	7563	2459	2289	2289
q6	194	171	136	136
q7	921	747	618	618
q8	9306	1322	1130	1130
q9	4869	4630	4666	4630
q10	6822	2327	1922	1922
q11	483	286	270	270
q12	354	382	226	226
q13	17772	3693	3110	3110
q14	224	232	202	202
q15	521	473	459	459
q16	621	631	573	573
q17	582	847	320	320
q18	6775	6086	6201	6086
q19	1675	967	570	570
q20	315	319	190	190
q21	2988	2198	1962	1962
q22	359	340	302	302
Total cold run time: 102654 ms
Total hot run time: 31613 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5184	5137	5137	5137
q2	233	327	231	231
q3	2113	2735	2269	2269
q4	1462	1804	1366	1366
q5	4241	4087	4127	4087
q6	213	172	129	129
q7	1842	1847	1711	1711
q8	2590	2565	2587	2565
q9	7171	7161	7068	7068
q10	3017	3208	2792	2792
q11	574	511	502	502
q12	686	748	614	614
q13	3545	3801	3278	3278
q14	289	298	272	272
q15	506	458	466	458
q16	630	679	635	635
q17	1133	1547	1387	1387
q18	7716	7281	7369	7281
q19	820	816	894	816
q20	2010	2030	1894	1894
q21	5443	4981	4989	4981
q22	599	592	540	540
Total cold run time: 52017 ms
Total hot run time: 50013 ms

dataroaring

LLGTM

github-actions · 2025-02-24T11:38:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-24T11:38:57Z

PR approved by anyone and no changes requested.

doris-robot · 2025-02-24T11:40:38Z

TPC-DS: Total hot run time: 190769 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc1301627bd1cfc23e1534519e1eef3b0f753f10, data reload: false

query1	1307	976	928	928
query2	6358	1831	1850	1831
query3	11072	4510	4470	4470
query4	25376	23863	23799	23799
query5	5501	686	518	518
query6	326	206	205	205
query7	3990	518	303	303
query8	299	280	245	245
query9	8530	2573	2571	2571
query10	510	321	267	267
query11	15768	15213	14964	14964
query12	159	109	109	109
query13	1547	523	369	369
query14	9693	6127	6215	6127
query15	209	188	181	181
query16	7599	697	475	475
query17	1096	733	588	588
query18	1995	432	339	339
query19	218	208	171	171
query20	128	128	119	119
query21	214	129	115	115
query22	4282	4691	4368	4368
query23	34374	33532	33247	33247
query24	8767	2497	2461	2461
query25	535	495	483	483
query26	1192	294	159	159
query27	2700	507	357	357
query28	4852	2442	2441	2441
query29	686	576	437	437
query30	220	189	162	162
query31	912	878	836	836
query32	72	67	67	67
query33	549	370	308	308
query34	782	885	518	518
query35	823	834	765	765
query36	974	1029	933	933
query37	150	93	76	76
query38	4185	4372	4241	4241
query39	1538	1475	1433	1433
query40	206	128	121	121
query41	51	50	55	50
query42	125	109	104	104
query43	512	501	471	471
query44	1336	798	809	798
query45	184	175	165	165
query46	901	1103	667	667
query47	1832	1839	1765	1765
query48	424	430	314	314
query49	807	499	427	427
query50	765	788	450	450
query51	4280	4301	4248	4248
query52	118	109	99	99
query53	239	274	192	192
query54	513	504	411	411
query55	82	83	85	83
query56	292	277	274	274
query57	1155	1169	1090	1090
query58	256	265	238	238
query59	2710	2701	2645	2645
query60	311	282	298	282
query61	118	112	115	112
query62	776	726	679	679
query63	251	206	200	200
query64	4091	1061	656	656
query65	3362	3264	3273	3264
query66	1393	400	299	299
query67	15860	15553	15328	15328
query68	8358	781	503	503
query69	469	297	277	277
query70	1243	1151	1092	1092
query71	398	305	263	263
query72	5905	3675	3706	3675
query73	671	767	345	345
query74	9256	9134	9048	9048
query75	3182	3158	2659	2659
query76	3063	1174	737	737
query77	475	383	278	278
query78	10095	9930	9244	9244
query79	2815	831	588	588
query80	1224	520	464	464
query81	537	276	239	239
query82	773	136	100	100
query83	169	167	151	151
query84	239	99	75	75
query85	766	347	371	347
query86	422	314	293	293
query87	4365	4573	4479	4479
query88	4352	2211	2187	2187
query89	425	305	289	289
query90	1921	192	191	191
query91	136	142	109	109
query92	79	57	54	54
query93	2655	990	557	557
query94	743	415	308	308
query95	346	272	260	260
query96	512	541	284	284
query97	2855	2848	2697	2697
query98	228	208	198	198
query99	1344	1406	1360	1360
Total cold run time: 282459 ms
Total hot run time: 190769 ms

doris-robot · 2025-02-24T11:45:56Z

ClickBench: Total hot run time: 30.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc1301627bd1cfc23e1534519e1eef3b0f753f10, data reload: false

query1	0.04	0.05	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.10
query5	0.40	0.41	0.40
query6	1.16	0.64	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.60	0.51	0.54
query10	0.57	0.58	0.57
query11	0.14	0.10	0.11
query12	0.14	0.10	0.12
query13	0.61	0.61	0.60
query14	2.70	2.68	2.66
query15	0.91	0.87	0.87
query16	0.37	0.37	0.38
query17	1.00	1.04	1.05
query18	0.21	0.20	0.19
query19	1.90	1.78	1.98
query20	0.01	0.01	0.01
query21	15.36	0.94	0.54
query22	0.75	1.25	0.64
query23	14.92	1.38	0.61
query24	7.24	1.18	1.00
query25	0.49	0.14	0.13
query26	0.54	0.16	0.13
query27	0.05	0.05	0.06
query28	9.87	0.85	0.44
query29	12.60	4.01	3.29
query30	0.25	0.09	0.06
query31	2.84	0.59	0.39
query32	3.23	0.54	0.46
query33	2.96	3.03	3.07
query34	15.86	5.15	4.53
query35	4.54	4.56	4.57
query36	0.66	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.44 s
Total hot run time: 30.75 s

PR #47416 already filters the staled responses by the job ID. But the task is getting by `(backendId, taskType, signature)`, so some staled responses might not be able to be filtered, if the two signatures are equal.

PR apache#47416 already filters the staled responses by the job ID. But the task is getting by `(backendId, taskType, signature)`, so some staled responses might not be able to be filtered, if the two signatures are equal.

…8238 (#48303) cherry pick from #48238

…8238 (#48291) Cherry-picked from #48238 Co-authored-by: walter <maochuan@selectdb.com>

PR apache#47416 already filters the staled responses by the job ID. But the task is getting by `(backendId, taskType, signature)`, so some staled responses might not be able to be filtered, if the two signatures are equal.

w41ter added dev/2.1.x dev/3.0.x labels Feb 24, 2025

[fix](backup) make snapshot task with unique signature

cc13016

to filter the staled responses

w41ter force-pushed the snapshot_task_with_unique_id branch from a9deac0 to cc13016 Compare February 24, 2025 10:44

dataroaring approved these changes Feb 24, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 24, 2025

github-actions bot added the reviewed label Feb 24, 2025

wyxxxcat approved these changes Feb 25, 2025

View reviewed changes

w41ter merged commit 6bcc797 into apache:master Feb 25, 2025
24 of 26 checks passed

w41ter deleted the snapshot_task_with_unique_id branch February 25, 2025 03:44

github-actions bot mentioned this pull request Feb 25, 2025

branch-3.0: [fix](backup) make snapshot task with unique signature #48238 #48291

Merged

github-actions bot added the dev/2.1.x-conflict label Feb 25, 2025

w41ter added the p0_w label Feb 25, 2025

w41ter mentioned this pull request Feb 25, 2025

branch-2.1: [fix](backup) make snapshot task with unique signature #48238 #48303

Merged

w41ter removed the dev/2.1.x-conflict label Feb 25, 2025

yiguolei pushed a commit that referenced this pull request Feb 27, 2025

branch-2.1: [fix](backup) make snapshot task with unique signature #4…

5f35bdf

…8238 (#48303) cherry pick from #48238

yiguolei removed the dev/2.1.x label Feb 27, 2025

yiguolei added the dev/2.1.9-merged label Feb 27, 2025

dataroaring pushed a commit that referenced this pull request Feb 27, 2025

branch-3.0: [fix](backup) make snapshot task with unique signature #4…

526f2c2

…8238 (#48291) Cherry-picked from #48238 Co-authored-by: walter <maochuan@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Feb 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](backup) make snapshot task with unique signature#48238

[fix](backup) make snapshot task with unique signature#48238
w41ter merged 1 commit intoapache:masterfrom
w41ter:snapshot_task_with_unique_id

w41ter commented Feb 24, 2025

Uh oh!

w41ter commented Feb 24, 2025

Uh oh!

Thearas commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

w41ter commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

dataroaring left a comment

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

w41ter commented Feb 24, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

w41ter commented Feb 24, 2025

Uh oh!

Thearas commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

w41ter commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

github-actions bot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

doris-robot commented Feb 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants