Skip to content

branch-3.0: [improve](statistic)Reduce call of olap table getVisibleVersion for cloud version. (#53903)#54661

Merged
dataroaring merged 1 commit intoapache:branch-3.0from
Jibing-Li:version3.0
Aug 14, 2025
Merged

branch-3.0: [improve](statistic)Reduce call of olap table getVisibleVersion for cloud version. (#53903)#54661
dataroaring merged 1 commit intoapache:branch-3.0from
Jibing-Li:version3.0

Conversation

@Jibing-Li
Copy link
Contributor

backport: #53903

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review August 13, 2025 06:45
@Jibing-Li Jibing-Li requested a review from dataroaring as a code owner August 13, 2025 06:45
@Jibing-Li
Copy link
Contributor Author

run buildall

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39730 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7b05253268950ea71345e56966703ed5c0828ef, data reload: false

------ Round 1 ----------------------------------
q1	17611	6721	6684	6684
q2	2064	226	173	173
q3	10475	1133	1182	1133
q4	10222	775	719	719
q5	7734	2885	2825	2825
q6	213	129	133	129
q7	969	640	629	629
q8	9370	1942	2023	1942
q9	6569	6393	6380	6380
q10	7041	2271	2325	2271
q11	458	272	262	262
q12	407	220	231	220
q13	18017	3012	2993	2993
q14	238	215	222	215
q15	509	462	464	462
q16	488	384	374	374
q17	982	525	564	525
q18	7216	6525	6706	6525
q19	1455	1068	1029	1029
q20	471	196	207	196
q21	3905	3139	3064	3064
q22	1102	981	980	980
Total cold run time: 107516 ms
Total hot run time: 39730 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6598	6554	6576	6554
q2	331	232	228	228
q3	2907	2888	2935	2888
q4	2045	1839	1810	1810
q5	5674	5734	5713	5713
q6	206	131	131	131
q7	2254	1821	1828	1821
q8	3371	3483	3554	3483
q9	8739	8937	8836	8836
q10	3599	3525	3518	3518
q11	607	478	488	478
q12	808	592	590	590
q13	8813	3196	3155	3155
q14	304	266	273	266
q15	504	471	459	459
q16	501	448	437	437
q17	1829	1626	1603	1603
q18	8402	7701	7674	7674
q19	1708	1529	1485	1485
q20	2056	1815	1810	1810
q21	5334	5099	5039	5039
q22	1145	1080	1021	1021
Total cold run time: 67735 ms
Total hot run time: 58999 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197379 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7b05253268950ea71345e56966703ed5c0828ef, data reload: false

query1	1294	978	913	913
query2	6261	1931	1903	1903
query3	10988	4340	4588	4340
query4	33382	23769	23914	23769
query5	4300	466	455	455
query6	274	178	177	177
query7	3982	311	323	311
query8	306	232	229	229
query9	9516	2590	2579	2579
query10	470	255	259	255
query11	18109	15157	15103	15103
query12	156	112	103	103
query13	1551	442	448	442
query14	9382	7142	6990	6990
query15	249	194	184	184
query16	7995	488	533	488
query17	1579	604	590	590
query18	2127	326	324	324
query19	254	171	164	164
query20	130	114	112	112
query21	203	105	111	105
query22	4614	4478	4411	4411
query23	35834	34353	34315	34315
query24	11287	2875	2938	2875
query25	647	400	407	400
query26	1232	168	165	165
query27	2848	339	338	338
query28	7726	2141	2128	2128
query29	854	443	435	435
query30	261	163	154	154
query31	1066	841	842	841
query32	96	55	60	55
query33	772	303	307	303
query34	939	511	551	511
query35	877	744	745	744
query36	1114	969	939	939
query37	124	63	65	63
query38	4085	3917	4035	3917
query39	1587	1485	1487	1485
query40	210	101	97	97
query41	51	51	46	46
query42	123	110	104	104
query43	519	478	466	466
query44	1251	817	813	813
query45	190	170	172	170
query46	1166	723	726	723
query47	2023	1877	1921	1877
query48	472	380	376	376
query49	956	405	440	405
query50	838	456	432	432
query51	7275	7369	7309	7309
query52	98	90	90	90
query53	262	186	184	184
query54	1150	478	488	478
query55	81	81	81	81
query56	271	240	244	240
query57	1327	1233	1202	1202
query58	228	226	228	226
query59	3178	2906	3008	2906
query60	303	269	260	260
query61	151	111	108	108
query62	840	711	675	675
query63	221	198	201	198
query64	4139	691	658	658
query65	3434	3320	3283	3283
query66	1001	296	296	296
query67	15834	15594	15722	15594
query68	4368	575	571	571
query69	425	264	266	264
query70	1114	1125	1088	1088
query71	333	250	262	250
query72	6306	4093	4171	4093
query73	752	354	356	354
query74	10137	8998	8936	8936
query75	3398	2678	2665	2665
query76	2597	1026	1006	1006
query77	376	286	280	280
query78	10536	9607	9541	9541
query79	2029	603	615	603
query80	893	452	452	452
query81	548	222	228	222
query82	744	97	89	89
query83	303	142	152	142
query84	233	87	79	79
query85	1529	315	297	297
query86	462	301	278	278
query87	4384	4319	4244	4244
query88	4246	2384	2368	2368
query89	417	297	288	288
query90	1866	190	189	189
query91	184	149	150	149
query92	66	53	50	50
query93	2464	551	548	548
query94	750	306	284	284
query95	362	256	259	256
query96	620	282	295	282
query97	3322	3123	3179	3123
query98	225	214	196	196
query99	1504	1280	1351	1280
Total cold run time: 303556 ms
Total hot run time: 197379 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7b05253268950ea71345e56966703ed5c0828ef, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.62	0.10	0.11
query5	0.52	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.56	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.59
query14	0.77	0.81	0.80
query15	0.86	0.84	0.83
query16	0.39	0.40	0.38
query17	1.07	1.07	1.04
query18	0.25	0.23	0.23
query19	1.86	1.82	1.74
query20	0.01	0.00	0.01
query21	15.42	0.55	0.58
query22	2.82	2.72	1.84
query23	16.88	0.98	0.89
query24	3.36	1.48	0.71
query25	0.27	0.19	0.13
query26	0.27	0.15	0.14
query27	0.04	0.05	0.05
query28	10.04	0.52	0.50
query29	12.62	3.16	3.19
query30	0.25	0.06	0.07
query31	2.86	0.39	0.38
query32	3.23	0.46	0.46
query33	2.97	3.07	3.04
query34	17.03	4.54	4.56
query35	4.58	4.58	4.54
query36	0.65	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.62 s
Total hot run time: 30.11 s

@Jibing-Li
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 63.64% (35/55) 🎉
Increment coverage report
Complete coverage report

@Jibing-Li
Copy link
Contributor Author

run buildall

…loud version. (apache#53903)

1. Reduce call of olap table getVisibleVersion for cloud version,
because it is an RPC in cloud mode.
2. Improve auto collect job append logic, make very low level jobs could
be executed more immediately
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39804 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32a4aa7628226611e39338776bf3b618148617a1, data reload: false

------ Round 1 ----------------------------------
q1	17574	6925	6669	6669
q2	2078	182	158	158
q3	10533	1109	1164	1109
q4	10216	748	685	685
q5	7682	2862	2820	2820
q6	209	133	129	129
q7	952	619	605	605
q8	9352	1943	1961	1943
q9	6642	6374	6387	6374
q10	7023	2268	2253	2253
q11	458	249	264	249
q12	407	213	216	213
q13	17828	2970	2972	2970
q14	235	199	210	199
q15	520	465	456	456
q16	480	393	368	368
q17	977	583	576	576
q18	7144	6678	6667	6667
q19	1449	1090	1078	1078
q20	473	196	204	196
q21	4084	3132	3107	3107
q22	1083	980	1003	980
Total cold run time: 107399 ms
Total hot run time: 39804 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6674	6685	6694	6685
q2	327	237	235	235
q3	2890	2935	2965	2935
q4	2047	1808	1823	1808
q5	5736	5687	5681	5681
q6	209	125	125	125
q7	2227	1777	1822	1777
q8	3331	3521	3533	3521
q9	8752	8867	8818	8818
q10	3590	3501	3504	3501
q11	607	488	482	482
q12	819	609	595	595
q13	9023	3220	3157	3157
q14	323	277	271	271
q15	506	463	464	463
q16	493	435	454	435
q17	1835	1621	1619	1619
q18	8197	7783	7925	7783
q19	1693	1400	1528	1400
q20	2067	1819	1858	1819
q21	5261	5062	5136	5062
q22	1106	1095	1065	1065
Total cold run time: 67713 ms
Total hot run time: 59237 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195756 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32a4aa7628226611e39338776bf3b618148617a1, data reload: false

query1	1283	923	918	918
query2	6345	1889	1842	1842
query3	10929	4386	4531	4386
query4	33440	23292	23757	23292
query5	3639	453	434	434
query6	251	193	174	174
query7	3993	314	318	314
query8	283	223	220	220
query9	9558	2560	2531	2531
query10	478	255	244	244
query11	17897	15170	15246	15170
query12	163	101	101	101
query13	1558	418	423	418
query14	9427	6520	7054	6520
query15	239	185	200	185
query16	8097	505	490	490
query17	1652	616	597	597
query18	2103	333	326	326
query19	241	173	183	173
query20	133	118	113	113
query21	203	109	112	109
query22	4955	4346	4450	4346
query23	35083	34157	34105	34105
query24	12265	2920	2983	2920
query25	708	424	435	424
query26	1778	171	178	171
query27	2902	352	369	352
query28	7889	2166	2161	2161
query29	1072	464	469	464
query30	282	159	163	159
query31	1094	829	869	829
query32	91	57	53	53
query33	766	290	290	290
query34	1051	501	537	501
query35	901	732	723	723
query36	1113	959	926	926
query37	257	70	68	68
query38	4186	3917	3959	3917
query39	1525	1486	1476	1476
query40	256	101	103	101
query41	53	48	47	47
query42	114	102	102	102
query43	518	477	461	461
query44	1297	836	834	834
query45	192	170	172	170
query46	1178	726	711	711
query47	2061	1874	1953	1874
query48	466	374	391	374
query49	1073	393	390	390
query50	850	452	438	438
query51	7501	7332	7143	7143
query52	106	86	91	86
query53	258	185	189	185
query54	1314	463	464	463
query55	75	79	79	79
query56	267	247	239	239
query57	1316	1211	1192	1192
query58	221	226	218	218
query59	3255	3073	2963	2963
query60	280	267	253	253
query61	110	110	114	110
query62	876	710	698	698
query63	227	194	189	189
query64	4844	651	644	644
query65	3363	3268	3306	3268
query66	1349	297	289	289
query67	15828	15528	15726	15528
query68	4788	573	558	558
query69	440	260	267	260
query70	1195	1084	1142	1084
query71	374	253	259	253
query72	6147	4053	3908	3908
query73	740	344	351	344
query74	10452	9298	9009	9009
query75	3377	2609	2627	2609
query76	2667	1123	1051	1051
query77	372	271	278	271
query78	10542	9679	9487	9487
query79	2596	605	608	605
query80	946	419	418	418
query81	544	215	220	215
query82	900	90	85	85
query83	244	145	146	145
query84	241	78	82	78
query85	1289	306	285	285
query86	453	296	297	296
query87	4366	4297	4257	4257
query88	4278	2386	2365	2365
query89	410	297	297	297
query90	2040	183	183	183
query91	182	143	148	143
query92	65	47	52	47
query93	2327	551	540	540
query94	884	304	265	265
query95	348	254	255	254
query96	617	282	276	276
query97	3283	3191	3196	3191
query98	216	201	193	193
query99	1513	1293	1304	1293
Total cold run time: 307914 ms
Total hot run time: 195756 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32a4aa7628226611e39338776bf3b618148617a1, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.52	0.51	0.52
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.50
query10	0.56	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	0.76	0.80	0.83
query15	0.85	0.82	0.84
query16	0.38	0.37	0.38
query17	1.02	1.06	1.08
query18	0.24	0.21	0.21
query19	1.90	1.87	1.81
query20	0.02	0.01	0.01
query21	15.36	0.58	0.58
query22	2.51	1.83	1.85
query23	16.93	0.90	0.72
query24	3.11	1.70	1.52
query25	0.30	0.08	0.06
query26	0.47	0.14	0.13
query27	0.03	0.04	0.04
query28	9.26	0.52	0.54
query29	12.58	3.22	3.19
query30	0.26	0.09	0.08
query31	2.81	0.40	0.40
query32	3.23	0.46	0.47
query33	3.01	3.03	3.02
query34	17.05	4.55	4.50
query35	4.64	4.63	4.56
query36	0.66	0.47	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.57 s
Total hot run time: 30.84 s

@Jibing-Li
Copy link
Contributor Author

run feut

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 63.64% (35/55) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

run nonConcurrent

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ddd4a74 into apache:branch-3.0 Aug 14, 2025
23 of 25 checks passed
@Jibing-Li Jibing-Li deleted the version3.0 branch August 14, 2025 03:45
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants