Skip to content

[enhancement](cloud) make file cache version upgrade faster #50726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
May 23, 2025
Merged

[enhancement](cloud) make file cache version upgrade faster #50726

merged 11 commits into from
May 23, 2025

Conversation

freemandealer
Copy link
Contributor

parallel the upgrade for each directory to shorten be restart time

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[enhancement](cloud) make file cache version upgrade faster
53f2449 
parallel the upgrade for each directory to shorten be restart time

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

run buildall

@freemandealer
fix _init_mem_env problem
200c0fc 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

run buildall

@freemandealer
minor fix for ut
a46e6c3 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@doris-robot
Copy link

TPC-H: Total hot run time: 33738 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 200c0fcc6ffabfded50abb1b408f9cfb7b5cbdd0, data reload: false

------ Round 1 ----------------------------------
q1	26574	5057	4978	4978
q2	2084	281	189	189
q3	10389	1243	681	681
q4	10234	1008	530	530
q5	7534	2351	2311	2311
q6	183	162	132	132
q7	928	747	625	625
q8	9315	1254	1044	1044
q9	6866	5107	5089	5089
q10	6876	2295	1890	1890
q11	507	282	272	272
q12	345	349	214	214
q13	17759	3692	3015	3015
q14	230	230	226	226
q15	516	488	482	482
q16	446	428	379	379
q17	673	853	380	380
q18	7401	7096	7079	7079
q19	1550	947	542	542
q20	342	330	216	216
q21	3994	3388	2507	2507
q22	1013	1001	957	957
Total cold run time: 115759 ms
Total hot run time: 33738 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5134	5054	5065	5054
q2	228	321	228	228
q3	2152	2615	2276	2276
q4	1353	1757	1330	1330
q5	4421	4377	4365	4365
q6	221	176	132	132
q7	2040	1915	1772	1772
q8	2598	2556	2560	2556
q9	7138	7188	7039	7039
q10	3043	3155	2716	2716
q11	569	507	476	476
q12	681	766	610	610
q13	3481	3815	3256	3256
q14	298	306	306	306
q15	533	486	476	476
q16	451	466	439	439
q17	1130	1531	1385	1385
q18	7723	7489	7432	7432
q19	815	827	861	827
q20	2011	1958	1848	1848
q21	5044	4783	4727	4727
q22	1088	1050	993	993
Total cold run time: 52152 ms
Total hot run time: 50243 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192110 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 200c0fcc6ffabfded50abb1b408f9cfb7b5cbdd0, data reload: false

query1	1398	1102	1055	1055
query2	6340	1840	1783	1783
query3	11003	4554	4522	4522
query4	51622	25754	23203	23203
query5	5148	602	443	443
query6	341	202	203	202
query7	4876	496	270	270
query8	302	254	240	240
query9	5433	2594	2585	2585
query10	434	319	261	261
query11	14951	15003	14820	14820
query12	158	113	113	113
query13	1039	530	405	405
query14	10104	6163	6296	6163
query15	209	205	181	181
query16	7131	673	504	504
query17	1100	729	595	595
query18	1561	419	320	320
query19	208	201	175	175
query20	130	133	129	129
query21	210	135	104	104
query22	4450	4512	4406	4406
query23	34028	33486	33109	33109
query24	6643	2385	2450	2385
query25	454	459	409	409
query26	717	271	155	155
query27	2302	528	336	336
query28	3032	2118	2095	2095
query29	589	578	443	443
query30	288	230	197	197
query31	843	879	784	784
query32	68	69	69	69
query33	508	367	311	311
query34	773	865	513	513
query35	802	818	759	759
query36	949	1003	902	902
query37	119	101	74	74
query38	4141	4182	4187	4182
query39	1575	1427	1445	1427
query40	208	120	108	108
query41	55	54	52	52
query42	122	112	114	112
query43	505	519	493	493
query44	1298	803	824	803
query45	186	173	166	166
query46	870	1043	632	632
query47	1864	1894	1848	1848
query48	383	432	316	316
query49	685	484	435	435
query50	669	755	416	416
query51	4232	4230	4212	4212
query52	107	112	102	102
query53	233	259	183	183
query54	578	572	530	530
query55	88	82	81	81
query56	317	295	282	282
query57	1164	1177	1151	1151
query58	263	258	247	247
query59	2672	2777	2766	2766
query60	328	334	298	298
query61	138	136	127	127
query62	761	767	671	671
query63	233	198	189	189
query64	1968	1065	714	714
query65	4389	4294	4224	4224
query66	753	405	307	307
query67	15947	15670	15398	15398
query68	6504	902	521	521
query69	535	300	274	274
query70	1188	1141	1119	1119
query71	499	312	282	282
query72	6019	4870	5081	4870
query73	1524	729	346	346
query74	8911	9145	8817	8817
query75	4045	3205	2716	2716
query76	4208	1177	749	749
query77	761	381	287	287
query78	9892	10024	9224	9224
query79	2429	809	564	564
query80	680	527	437	437
query81	481	255	223	223
query82	444	125	97	97
query83	301	250	235	235
query84	305	102	90	90
query85	775	366	323	323
query86	374	301	296	296
query87	4393	4451	4269	4269
query88	3476	2152	2198	2152
query89	408	314	284	284
query90	1862	206	206	206
query91	220	139	110	110
query92	81	56	55	55
query93	2003	950	577	577
query94	665	410	307	307
query95	369	290	292	290
query96	508	568	274	274
query97	3166	3242	3153	3153
query98	248	208	200	200
query99	1430	1383	1271	1271
Total cold run time: 295699 ms
Total hot run time: 192110 ms

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

ClickBench: Total hot run time: 29.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 200c0fcc6ffabfded50abb1b408f9cfb7b5cbdd0, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.60	0.59
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.53	0.53
query10	0.57	0.59	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.59	0.59
query14	0.78	0.80	0.80
query15	0.87	0.84	0.86
query16	0.38	0.37	0.39
query17	1.07	1.06	1.06
query18	0.21	0.20	0.20
query19	1.89	1.78	1.85
query20	0.02	0.01	0.01
query21	15.40	0.87	0.56
query22	0.75	1.24	0.67
query23	14.89	1.39	0.67
query24	6.94	2.93	0.49
query25	0.48	0.31	0.12
query26	0.60	0.17	0.14
query27	0.05	0.05	0.05
query28	9.97	0.77	0.45
query29	12.54	3.89	3.26
query30	0.25	0.09	0.06
query31	2.84	0.59	0.37
query32	3.24	0.54	0.47
query33	3.08	3.06	3.08
query34	16.07	5.04	4.53
query35	4.53	4.50	4.49
query36	0.67	0.50	0.47
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.92 s
Total hot run time: 29.01 s

@doris-robot
Copy link

TPC-H: Total hot run time: 34037 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a46e6c371a43024e70c7a1a7faf6a80396d2ecf8, data reload: false

------ Round 1 ----------------------------------
q1	26771	5085	5048	5048
q2	2080	280	183	183
q3	10397	1265	671	671
q4	10229	1009	522	522
q5	7556	2349	2357	2349
q6	185	160	132	132
q7	909	729	630	630
q8	9305	1296	1146	1146
q9	6745	5087	5124	5087
q10	6814	2304	1886	1886
q11	503	288	289	288
q12	340	356	210	210
q13	17793	3691	3087	3087
q14	235	226	213	213
q15	517	480	475	475
q16	428	434	377	377
q17	611	842	385	385
q18	8090	7248	7164	7164
q19	1226	958	567	567
q20	347	339	229	229
q21	3980	3413	2422	2422
q22	1027	1030	966	966
Total cold run time: 116088 ms
Total hot run time: 34037 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5115	5109	5047	5047
q2	232	331	227	227
q3	2177	2663	2270	2270
q4	1352	1797	1374	1374
q5	4194	4100	4141	4100
q6	204	179	125	125
q7	1878	1808	1645	1645
q8	2472	2519	2413	2413
q9	6774	6714	6603	6603
q10	2864	3084	2647	2647
q11	568	485	476	476
q12	632	718	618	618
q13	3406	3703	3107	3107
q14	265	276	273	273
q15	515	475	473	473
q16	425	476	429	429
q17	1098	1491	1319	1319
q18	7209	7062	7165	7062
q19	803	821	890	821
q20	1913	1962	1835	1835
q21	4963	4660	4670	4660
q22	1064	997	973	973
Total cold run time: 50123 ms
Total hot run time: 48497 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185900 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a46e6c371a43024e70c7a1a7faf6a80396d2ecf8, data reload: false

query1	998	480	512	480
query2	6563	1949	1938	1938
query3	6734	233	216	216
query4	25923	23865	23590	23590
query5	5806	624	461	461
query6	309	205	193	193
query7	4648	496	289	289
query8	311	256	241	241
query9	8657	2560	2568	2560
query10	541	323	264	264
query11	15436	15566	14831	14831
query12	161	113	111	111
query13	1685	536	404	404
query14	10907	6119	6085	6085
query15	250	198	176	176
query16	7793	661	448	448
query17	1562	747	582	582
query18	2035	406	306	306
query19	267	188	181	181
query20	122	120	112	112
query21	219	124	113	113
query22	4055	4134	4013	4013
query23	33789	32876	32888	32876
query24	8080	2397	2411	2397
query25	504	470	383	383
query26	1210	269	153	153
query27	2710	491	335	335
query28	4376	2134	2135	2134
query29	700	543	426	426
query30	286	223	183	183
query31	917	863	774	774
query32	76	63	61	61
query33	545	389	297	297
query34	796	837	524	524
query35	799	804	718	718
query36	988	1040	894	894
query37	112	101	79	79
query38	4141	4138	4203	4138
query39	1449	1391	1393	1391
query40	209	121	106	106
query41	57	56	54	54
query42	120	110	106	106
query43	507	544	484	484
query44	1282	792	796	792
query45	180	174	169	169
query46	849	1019	625	625
query47	1725	1779	1714	1714
query48	369	411	311	311
query49	752	488	419	419
query50	665	677	405	405
query51	4118	4052	4095	4052
query52	104	102	102	102
query53	227	246	189	189
query54	586	587	494	494
query55	79	82	83	82
query56	309	302	288	288
query57	1110	1145	1093	1093
query58	283	257	259	257
query59	2756	2828	2666	2666
query60	329	315	312	312
query61	132	130	130	130
query62	762	712	656	656
query63	236	186	183	183
query64	4225	1018	667	667
query65	4291	4210	4250	4210
query66	1017	414	308	308
query67	15620	15610	15335	15335
query68	8157	891	511	511
query69	505	297	257	257
query70	1244	1110	1122	1110
query71	512	326	286	286
query72	5784	4714	4749	4714
query73	720	627	355	355
query74	8819	9076	8688	8688
query75	4004	3208	2708	2708
query76	3760	1182	754	754
query77	780	384	281	281
query78	9896	10050	9294	9294
query79	2741	817	580	580
query80	613	519	432	432
query81	496	249	220	220
query82	481	134	95	95
query83	276	249	247	247
query84	303	104	88	88
query85	760	353	308	308
query86	355	307	285	285
query87	4391	4307	4262	4262
query88	3649	2213	2211	2211
query89	384	321	278	278
query90	1939	205	209	205
query91	148	149	123	123
query92	74	60	58	58
query93	1930	974	579	579
query94	648	397	298	298
query95	369	291	280	280
query96	502	570	274	274
query97	3188	3243	3096	3096
query98	243	210	206	206
query99	1445	1382	1251	1251
Total cold run time: 278475 ms
Total hot run time: 185900 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a46e6c371a43024e70c7a1a7faf6a80396d2ecf8, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.11	0.11
query3	0.26	0.20	0.19
query4	1.60	0.18	0.18
query5	0.61	0.57	0.58
query6	1.19	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.06	0.03
query9	0.57	0.54	0.52
query10	0.58	0.57	0.59
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.61
query14	0.78	0.81	0.81
query15	0.87	0.85	0.86
query16	0.37	0.38	0.40
query17	1.04	1.01	1.06
query18	0.21	0.20	0.19
query19	1.88	1.79	1.82
query20	0.01	0.01	0.01
query21	15.40	0.87	0.55
query22	0.76	1.21	0.68
query23	14.92	1.40	0.64
query24	6.89	1.67	0.47
query25	0.47	0.24	0.06
query26	0.67	0.16	0.13
query27	0.05	0.05	0.05
query28	8.85	0.87	0.43
query29	12.56	3.94	3.25
query30	0.26	0.09	0.07
query31	2.81	0.59	0.38
query32	3.22	0.55	0.46
query33	3.04	3.05	3.12
query34	15.75	5.10	4.45
query35	4.56	4.54	4.51
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.48 s
Total hot run time: 28.78 s

@freemandealer
Copy link
Contributor Author

run cloud_p0

@freemandealer
Copy link
Contributor Author

run beut

@freemandealer
fix beut
0732e55 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33937 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0732e55b0e235899c8af1b833960ad1e917387ae, data reload: false

------ Round 1 ----------------------------------
q1	26001	5019	4999	4999
q2	2065	293	188	188
q3	10376	1310	705	705
q4	10223	1009	534	534
q5	7516	2288	2404	2288
q6	184	167	137	137
q7	928	767	618	618
q8	9317	1347	1098	1098
q9	6802	5110	5039	5039
q10	6886	2317	1924	1924
q11	485	290	271	271
q12	348	348	216	216
q13	17769	3656	3106	3106
q14	229	227	214	214
q15	535	485	487	485
q16	429	433	373	373
q17	611	864	353	353
q18	7728	7161	7170	7161
q19	1477	955	572	572
q20	330	346	219	219
q21	3852	3340	2455	2455
q22	1083	1033	982	982
Total cold run time: 115174 ms
Total hot run time: 33937 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5141	5108	5109	5108
q2	240	325	224	224
q3	2186	2662	2305	2305
q4	1388	1824	1377	1377
q5	4429	4383	4362	4362
q6	216	178	129	129
q7	2059	1929	1752	1752
q8	2556	2529	2505	2505
q9	7326	7320	7002	7002
q10	3050	3205	2721	2721
q11	578	524	497	497
q12	706	755	617	617
q13	3514	3872	3271	3271
q14	296	327	260	260
q15	544	493	485	485
q16	433	481	457	457
q17	1161	1596	1368	1368
q18	7949	7512	7432	7432
q19	816	829	976	829
q20	1962	1960	1928	1928
q21	5195	4712	4759	4712
q22	1130	1046	1036	1036
Total cold run time: 52875 ms
Total hot run time: 50377 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193992 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0732e55b0e235899c8af1b833960ad1e917387ae, data reload: false

query1	1394	1083	1063	1063
query2	6165	1841	1824	1824
query3	11051	4519	4690	4519
query4	53545	24721	23378	23378
query5	5186	545	465	465
query6	379	213	195	195
query7	4946	525	298	298
query8	328	257	246	246
query9	5999	2661	2635	2635
query10	440	333	275	275
query11	15101	15070	15085	15070
query12	166	114	106	106
query13	1099	529	420	420
query14	10067	6277	6304	6277
query15	208	195	184	184
query16	7077	635	464	464
query17	1066	747	573	573
query18	1500	392	301	301
query19	198	195	162	162
query20	128	119	123	119
query21	209	134	105	105
query22	4488	4627	4476	4476
query23	34275	33557	33615	33557
query24	6674	2417	2427	2417
query25	481	465	415	415
query26	725	279	162	162
query27	2274	514	359	359
query28	3103	2158	2137	2137
query29	567	572	435	435
query30	273	224	189	189
query31	900	820	814	814
query32	74	66	66	66
query33	469	355	318	318
query34	761	888	559	559
query35	799	825	759	759
query36	951	1001	876	876
query37	117	103	77	77
query38	4306	4259	4156	4156
query39	1528	1446	1435	1435
query40	211	116	109	109
query41	58	61	53	53
query42	126	111	117	111
query43	509	502	498	498
query44	1313	845	829	829
query45	184	177	163	163
query46	868	1057	671	671
query47	1891	1948	1780	1780
query48	403	438	322	322
query49	691	536	446	446
query50	674	712	400	400
query51	4198	4195	4241	4195
query52	118	108	100	100
query53	232	260	187	187
query54	632	605	542	542
query55	86	86	84	84
query56	337	311	316	311
query57	1246	1256	1139	1139
query58	262	266	249	249
query59	2751	2779	2748	2748
query60	338	328	313	313
query61	137	133	125	125
query62	701	733	694	694
query63	225	192	187	187
query64	1828	1014	698	698
query65	4407	4303	4213	4213
query66	726	409	358	358
query67	16075	15821	15553	15553
query68	5765	880	527	527
query69	537	310	266	266
query70	1224	1126	1105	1105
query71	442	331	296	296
query72	6022	4836	4929	4836
query73	1258	682	350	350
query74	9027	8905	8806	8806
query75	3315	3253	2676	2676
query76	3947	1192	774	774
query77	528	423	292	292
query78	10037	10296	9450	9450
query79	1734	874	577	577
query80	665	519	434	434
query81	502	245	219	219
query82	391	131	97	97
query83	259	261	236	236
query84	297	99	84	84
query85	748	350	312	312
query86	349	316	295	295
query87	4453	4407	4382	4382
query88	2894	2307	2254	2254
query89	400	334	294	294
query90	1670	201	210	201
query91	141	140	110	110
query92	67	58	52	52
query93	2386	923	579	579
query94	635	411	302	302
query95	360	284	289	284
query96	486	557	278	278
query97	3116	3241	3142	3142
query98	244	223	204	204
query99	1303	1398	1314	1314
Total cold run time: 295446 ms
Total hot run time: 193992 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0732e55b0e235899c8af1b833960ad1e917387ae, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.12
query3	0.26	0.19	0.19
query4	1.60	0.20	0.20
query5	0.61	0.59	0.60
query6	1.19	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.53	0.51
query10	0.57	0.58	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	0.78	0.82	0.80
query15	0.88	0.85	0.86
query16	0.39	0.38	0.37
query17	1.04	1.01	1.04
query18	0.23	0.21	0.22
query19	1.90	1.81	1.81
query20	0.01	0.01	0.01
query21	15.39	0.86	0.53
query22	0.78	1.17	0.65
query23	14.97	1.37	0.57
query24	7.40	1.38	0.47
query25	0.50	0.24	0.05
query26	0.58	0.16	0.14
query27	0.05	0.05	0.05
query28	9.85	0.87	0.42
query29	12.59	3.90	3.24
query30	0.25	0.11	0.06
query31	2.81	0.59	0.37
query32	3.23	0.54	0.46
query33	3.09	3.05	3.15
query34	15.80	5.11	4.50
query35	4.48	4.49	4.49
query36	0.68	0.49	0.47
query37	0.08	0.06	0.07
query38	0.06	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.08 s
Total hot run time: 28.62 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 70.59% (12/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.79% (14895/26698)
Line Coverage 44.60% (131848/295606)
Region Coverage 43.66% (66294/151856)
Branch Coverage 38.28% (33976/88766)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 76.47% (13/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.46% (20869/26264)
Line Coverage 72.70% (214860/295542)
Region Coverage 70.88% (126410/178356)
Branch Coverage 64.57% (65455/101374)

@freemandealer
Copy link
Contributor Author

run p0

@freemandealer
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 76.47% (13/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.34% (20839/26264)
Line Coverage 72.55% (214424/295542)
Region Coverage 70.72% (126131/178356)
Branch Coverage 64.46% (65342/101374)

@freemandealer
fix rename failure
420e78d 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
minor fix
58ce2e3 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.16% (18672/28223)
Region Coverage 65.82% (9265/14077)
Branch Coverage 55.58% (4982/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 33829 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

------ Round 1 ----------------------------------
q1	26077	5025	5059	5025
q2	2088	290	189	189
q3	10489	1247	719	719
q4	10233	1019	517	517
q5	7673	2441	2331	2331
q6	182	164	129	129
q7	915	746	610	610
q8	9321	1271	1075	1075
q9	6829	5086	5079	5079
q10	6810	2313	1904	1904
q11	476	290	273	273
q12	357	349	210	210
q13	17800	3764	3098	3098
q14	237	228	215	215
q15	546	500	514	500
q16	427	448	364	364
q17	625	862	370	370
q18	7577	7311	7111	7111
q19	1822	949	573	573
q20	340	353	223	223
q21	3793	2560	2354	2354
q22	1054	1018	960	960
Total cold run time: 115671 ms
Total hot run time: 33829 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5238	5138	5126	5126
q2	241	319	226	226
q3	2189	2708	2304	2304
q4	1356	1759	1377	1377
q5	4589	4524	4402	4402
q6	214	171	122	122
q7	1975	1934	1746	1746
q8	2569	2469	2608	2469
q9	7219	6999	7121	6999
q10	3000	3198	2772	2772
q11	567	496	482	482
q12	674	757	594	594
q13	3530	3924	3354	3354
q14	274	289	278	278
q15	550	528	517	517
q16	433	478	447	447
q17	1176	1554	1396	1396
q18	7610	7458	7362	7362
q19	856	824	941	824
q20	1993	2006	1875	1875
q21	5031	4471	4347	4347
q22	1083	1044	970	970
Total cold run time: 52367 ms
Total hot run time: 49989 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

query1	1010	484	519	484
query2	6570	1902	1862	1862
query3	6764	230	225	225
query4	26223	23399	23698	23399
query5	4344	639	445	445
query6	290	209	220	209
query7	4623	520	281	281
query8	277	238	236	236
query9	8594	2597	2603	2597
query10	470	313	275	275
query11	15306	15045	14957	14957
query12	161	109	109	109
query13	1673	537	405	405
query14	8883	6219	6254	6219
query15	208	194	169	169
query16	7137	660	448	448
query17	1204	694	550	550
query18	1971	401	289	289
query19	186	184	157	157
query20	118	126	118	118
query21	215	123	105	105
query22	4045	4295	4008	4008
query23	34174	33094	33096	33094
query24	8466	2415	2415	2415
query25	540	461	398	398
query26	1239	291	156	156
query27	2738	521	349	349
query28	4313	2145	2111	2111
query29	777	556	428	428
query30	280	220	187	187
query31	924	836	740	740
query32	77	62	65	62
query33	553	378	311	311
query34	796	914	511	511
query35	806	835	750	750
query36	982	984	908	908
query37	115	103	76	76
query38	4104	4104	4075	4075
query39	1503	1410	1387	1387
query40	211	124	110	110
query41	60	61	54	54
query42	123	107	106	106
query43	523	511	500	500
query44	1345	811	800	800
query45	176	177	168	168
query46	890	1059	630	630
query47	1734	1785	1705	1705
query48	406	433	313	313
query49	784	501	421	421
query50	680	731	431	431
query51	4190	4141	4094	4094
query52	112	113	106	106
query53	234	281	184	184
query54	584	602	501	501
query55	84	82	94	82
query56	343	292	302	292
query57	1125	1132	1099	1099
query58	266	259	251	251
query59	2671	2634	2638	2634
query60	330	325	301	301
query61	134	128	167	128
query62	817	732	673	673
query63	228	196	187	187
query64	4336	1008	668	668
query65	4291	4192	4237	4192
query66	1141	410	315	315
query67	15786	15480	15405	15405
query68	4650	902	534	534
query69	491	326	267	267
query70	1218	1097	1121	1097
query71	414	322	300	300
query72	5759	4709	4709	4709
query73	650	606	349	349
query74	9186	9342	8873	8873
query75	3219	3194	2704	2704
query76	3153	1264	799	799
query77	482	385	300	300
query78	9959	10074	9423	9423
query79	1983	854	585	585
query80	662	554	488	488
query81	498	269	218	218
query82	218	130	101	101
query83	252	256	239	239
query84	250	114	84	84
query85	751	361	312	312
query86	375	312	303	303
query87	4353	4434	4290	4290
query88	2900	2250	2260	2250
query89	398	315	280	280
query90	1978	212	204	204
query91	136	144	112	112
query92	80	69	58	58
query93	2420	948	583	583
query94	686	420	302	302
query95	377	293	300	293
query96	507	643	290	290
query97	2645	2802	2643	2643
query98	226	232	207	207
query99	1305	1404	1320	1320
Total cold run time: 268444 ms
Total hot run time: 186372 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.26	0.20	0.20
query4	1.59	0.19	0.11
query5	0.45	0.42	0.42
query6	1.17	0.65	0.68
query7	0.03	0.02	0.01
query8	0.05	0.04	0.03
query9	0.59	0.51	0.52
query10	0.57	0.59	0.55
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.62	0.61	0.60
query14	0.78	0.82	0.80
query15	0.88	0.86	0.88
query16	0.39	0.39	0.39
query17	1.06	1.04	1.08
query18	0.22	0.21	0.22
query19	1.96	1.86	1.81
query20	0.01	0.01	0.01
query21	15.42	0.89	0.54
query22	0.77	1.21	0.65
query23	14.91	1.37	0.60
query24	6.67	1.52	0.97
query25	0.50	0.21	0.07
query26	0.47	0.17	0.15
query27	0.05	0.05	0.05
query28	10.12	0.92	0.44
query29	12.57	4.03	3.34
query30	0.25	0.08	0.07
query31	2.83	0.60	0.38
query32	3.23	0.56	0.48
query33	3.06	3.10	3.08
query34	15.78	5.15	4.49
query35	4.52	4.55	4.53
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.4 s
Total hot run time: 29.21 s

@freemandealer
Copy link
Contributor Author

run p0

@freemandealer
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.14% (18667/28223)
Region Coverage 65.81% (9264/14077)
Branch Coverage 55.57% (4981/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 33745 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

------ Round 1 ----------------------------------
q1	26148	5140	5000	5000
q2	2078	280	176	176
q3	10760	1251	714	714
q4	10280	1011	516	516
q5	8178	2358	2324	2324
q6	188	161	131	131
q7	950	725	625	625
q8	9310	1278	1050	1050
q9	6843	5074	5049	5049
q10	6851	2322	1892	1892
q11	506	298	279	279
q12	366	372	218	218
q13	17790	3651	3106	3106
q14	227	229	217	217
q15	533	493	485	485
q16	410	440	388	388
q17	595	858	354	354
q18	7614	7228	7103	7103
q19	1879	941	573	573
q20	328	329	235	235
q21	3745	2561	2368	2368
q22	1050	1018	942	942
Total cold run time: 116629 ms
Total hot run time: 33745 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5218	5015	5088	5015
q2	241	327	231	231
q3	2178	2667	2295	2295
q4	1339	1810	1419	1419
q5	4518	4375	4353	4353
q6	213	171	126	126
q7	2026	1879	1807	1807
q8	2569	2542	2587	2542
q9	7275	7247	6849	6849
q10	3035	3169	2744	2744
q11	561	532	498	498
q12	679	790	663	663
q13	3608	3905	3344	3344
q14	290	288	265	265
q15	517	491	487	487
q16	462	489	438	438
q17	1158	1600	1340	1340
q18	7768	7545	7492	7492
q19	811	896	1034	896
q20	1993	2093	1868	1868
q21	4856	4541	4566	4541
q22	1081	1041	1009	1009
Total cold run time: 52396 ms
Total hot run time: 50222 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192918 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

query1	1409	1107	1092	1092
query2	6199	1866	1860	1860
query3	11061	4603	4432	4432
query4	53262	24892	22971	22971
query5	4958	502	481	481
query6	339	207	199	199
query7	4904	548	302	302
query8	321	254	246	246
query9	5717	2658	2647	2647
query10	439	335	279	279
query11	14948	15182	14935	14935
query12	176	117	102	102
query13	1043	554	400	400
query14	10190	6338	6429	6338
query15	223	205	182	182
query16	7041	643	493	493
query17	1058	739	568	568
query18	1579	409	318	318
query19	197	193	164	164
query20	125	129	126	126
query21	204	125	107	107
query22	4407	4445	4399	4399
query23	34070	33621	33735	33621
query24	6614	2454	2428	2428
query25	484	483	420	420
query26	673	286	159	159
query27	2237	535	348	348
query28	3196	2159	2173	2159
query29	617	586	476	476
query30	270	230	205	205
query31	903	892	793	793
query32	83	70	67	67
query33	471	391	323	323
query34	776	853	532	532
query35	851	859	769	769
query36	948	1010	912	912
query37	132	119	79	79
query38	4231	4296	4265	4265
query39	1525	1477	1455	1455
query40	219	126	132	126
query41	63	60	57	57
query42	128	118	117	117
query43	525	522	494	494
query44	1391	840	825	825
query45	184	175	164	164
query46	852	1030	653	653
query47	1826	1864	1837	1837
query48	406	439	345	345
query49	720	534	470	470
query50	685	719	413	413
query51	4240	4259	4290	4259
query52	113	113	95	95
query53	231	262	189	189
query54	592	588	550	550
query55	92	90	94	90
query56	322	304	313	304
query57	1194	1211	1147	1147
query58	272	266	256	256
query59	2787	2794	2737	2737
query60	347	329	310	310
query61	129	122	143	122
query62	731	749	673	673
query63	226	188	181	181
query64	1702	1053	685	685
query65	4319	4292	4268	4268
query66	750	398	306	306
query67	15901	15758	15169	15169
query68	7401	886	515	515
query69	542	316	259	259
query70	1205	1136	1071	1071
query71	506	330	305	305
query72	5491	4874	4891	4874
query73	1448	662	349	349
query74	9188	9024	8837	8837
query75	3744	3187	2691	2691
query76	4256	1179	764	764
query77	594	375	280	280
query78	10148	10041	9376	9376
query79	4140	781	558	558
query80	635	527	461	461
query81	491	261	220	220
query82	525	129	94	94
query83	375	249	238	238
query84	302	105	87	87
query85	814	362	363	362
query86	434	296	274	274
query87	4423	4414	4333	4333
query88	3683	2268	2291	2268
query89	431	327	287	287
query90	1847	209	200	200
query91	142	140	108	108
query92	75	69	54	54
query93	3012	944	578	578
query94	666	386	286	286
query95	374	299	283	283
query96	499	576	281	281
query97	2715	2715	2635	2635
query98	230	209	203	203
query99	1429	1395	1293	1293
Total cold run time: 300593 ms
Total hot run time: 192918 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bac0a8ceea8b97de5d2abb97f1c0cbc8f5a83a12, data reload: false

query1	0.04	0.03	0.04
query2	0.15	0.11	0.10
query3	0.34	0.21	0.20
query4	1.59	0.20	0.07
query5	0.42	0.40	0.40
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.63	0.52	0.53
query10	0.57	0.60	0.56
query11	0.25	0.13	0.13
query12	0.26	0.14	0.14
query13	0.64	0.63	0.62
query14	0.80	0.83	0.83
query15	0.97	0.86	0.87
query16	0.37	0.36	0.37
query17	1.07	1.05	1.07
query18	0.17	0.18	0.18
query19	1.99	1.79	1.86
query20	0.02	0.02	0.01
query21	15.39	0.96	0.66
query22	0.92	1.02	0.77
query23	14.72	1.53	0.74
query24	4.72	0.65	0.33
query25	0.17	0.08	0.08
query26	0.57	0.22	0.18
query27	0.08	0.08	0.09
query28	11.03	1.23	0.57
query29	12.55	4.03	3.41
query30	0.28	0.09	0.07
query31	2.82	0.63	0.43
query32	3.24	0.60	0.51
query33	3.03	3.17	3.09
query34	16.29	5.13	4.40
query35	4.43	4.44	4.45
query36	0.65	0.50	0.49
query37	0.19	0.18	0.17
query38	0.17	0.16	0.17
query39	0.05	0.04	0.04
query40	0.19	0.17	0.16
query41	0.10	0.05	0.06
query42	0.07	0.05	0.05
query43	0.06	0.05	0.04
Total cold run time: 103.23 s
Total hot run time: 29.49 s

@freemandealer
Copy link
Contributor Author

run feut

@freemandealer
Copy link
Contributor Author

run beut

@freemandealer
Copy link
Contributor Author

run p0

gavinchou
gavinchou reviewed May 23, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 23, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@gavinchou gavinchou added the p0_b label May 23, 2025
@gavinchou gavinchou merged commit 5fdaf7b into apache:master May 23, 2025
25 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request May 23, 2025
@freemandealer
[enhancement](cloud) make file cache version upgrade faster (#50726)
f725755 
parallel the upgrade for each directory to shorten be restart time
@github-actions github-actions bot mentioned this pull request May 23, 2025
Merged
dataroaring pushed a commit that referenced this pull request May 28, 2025
@github-actions @freemandealer
branch-3.0: [enhancement](cloud) make file cache version upgrade faster
b43fb25 
#50726 (#51194)

Cherry-picked from #50726

Co-authored-by: zhengyu <zhangzhengyu@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@freemandealer @koarz
[enhancement](cloud) make file cache version upgrade faster (apache#5…
84d8021 
…0726)

parallel the upgrade for each directory to shorten be restart time
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@koarz koarz koarz approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.6-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@freemandealer @hello-stephen @doris-robot @gavinchou @koarz @dataroaring