Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(cloud-merge) Optimize the lru queue elimination strategy #34139

Merged
merged 2 commits into from
Apr 30, 2024

Conversation

Lchangliang
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

3140

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -20,6 +20,7 @@
#include <bvar/bvar.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'bvar/bvar.h' file not found [clang-diagnostic-error]

#include <bvar/bvar.h>
         ^

}
}

TEST_F(BlockFileCacheTest, remove_from_other_queue_2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]

    ^
Additional context

be/test/io/cache/block_file_cache_test.cpp:3714: 104 lines including whitespace and comments (threshold 80)

    ^

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}
}

TEST_F(BlockFileCacheTest, remove_from_other_queue_2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]

    ^
Additional context

be/test/io/cache/block_file_cache_test.cpp:3765: 104 lines including whitespace and comments (threshold 80)

    ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.19% (8919/25342)
Line Coverage: 26.98% (73363/271888)
Region Coverage: 26.16% (37906/144895)
Branch Coverage: 22.98% (19301/84000)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fab276013781e3a5124633f6ba913600af44b22b_fab276013781e3a5124633f6ba913600af44b22b/report/index.html

@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41558 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d50d9cce55188491e6c8516cf23b189b503b2d31, data reload: false

------ Round 1 ----------------------------------
q1	17746	4595	4389	4389
q2	2521	194	189	189
q3	10907	1193	1148	1148
q4	10627	758	785	758
q5	7573	2757	2714	2714
q6	219	135	138	135
q7	967	569	532	532
q8	9581	2116	2080	2080
q9	9939	7372	7275	7275
q10	8748	3749	3740	3740
q11	447	232	245	232
q12	527	226	228	226
q13	18549	2950	2987	2950
q14	277	237	246	237
q15	525	466	477	466
q16	508	382	380	380
q17	965	696	738	696
q18	7986	7389	7326	7326
q19	1610	1512	1509	1509
q20	654	311	302	302
q21	5088	4001	4111	4001
q22	346	290	273	273
Total cold run time: 116310 ms
Total hot run time: 41558 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4369	4305	4273	4273
q2	358	265	279	265
q3	3028	2723	2747	2723
q4	1888	1573	1571	1571
q5	5333	5317	5319	5317
q6	205	122	126	122
q7	1786	1427	1400	1400
q8	3211	3371	3412	3371
q9	9272	9271	9287	9271
q10	3939	3711	3721	3711
q11	587	482	470	470
q12	781	608	598	598
q13	16421	2928	2950	2928
q14	319	264	272	264
q15	508	462	480	462
q16	469	417	442	417
q17	1759	1483	1454	1454
q18	7683	7487	7350	7350
q19	1654	1563	1574	1563
q20	1936	1755	1791	1755
q21	5017	4781	4797	4781
q22	558	469	490	469
Total cold run time: 71081 ms
Total hot run time: 54535 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187096 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d50d9cce55188491e6c8516cf23b189b503b2d31, data reload: false

query1	900	361	344	344
query2	6470	2552	2393	2393
query3	6658	200	205	200
query4	23209	21573	21482	21482
query5	4129	415	407	407
query6	272	171	172	171
query7	4599	283	280	280
query8	253	180	188	180
query9	8514	2368	2354	2354
query10	589	256	248	248
query11	14653	14161	14201	14161
query12	142	92	87	87
query13	1637	356	355	355
query14	10668	7331	7599	7331
query15	256	184	185	184
query16	8181	257	262	257
query17	1919	574	545	545
query18	2117	284	270	270
query19	195	146	155	146
query20	95	85	84	84
query21	198	125	125	125
query22	5121	4897	4934	4897
query23	33843	33254	33240	33240
query24	10723	3040	2957	2957
query25	615	367	367	367
query26	1384	154	152	152
query27	3005	310	316	310
query28	7573	2000	1992	1992
query29	892	595	591	591
query30	294	152	150	150
query31	963	744	737	737
query32	116	52	54	52
query33	743	249	244	244
query34	1073	476	501	476
query35	813	686	689	686
query36	1070	906	896	896
query37	127	64	69	64
query38	3323	3190	3187	3187
query39	1567	1548	1544	1544
query40	200	125	127	125
query41	43	39	40	39
query42	103	96	94	94
query43	615	544	557	544
query44	1207	728	735	728
query45	283	256	244	244
query46	1086	736	737	736
query47	1967	1895	1888	1888
query48	373	285	285	285
query49	1087	406	389	389
query50	758	383	394	383
query51	6720	6619	6566	6566
query52	101	97	87	87
query53	343	279	282	279
query54	323	240	264	240
query55	83	76	76	76
query56	250	232	227	227
query57	1221	1132	1131	1131
query58	273	191	206	191
query59	3480	3242	3211	3211
query60	245	251	231	231
query61	90	88	87	87
query62	665	438	443	438
query63	304	276	279	276
query64	8573	7189	7124	7124
query65	3128	3061	3025	3025
query66	1393	349	331	331
query67	15452	14967	14942	14942
query68	5219	528	532	528
query69	478	305	291	291
query70	1156	1144	1116	1116
query71	392	269	267	267
query72	7723	2648	2428	2428
query73	714	318	324	318
query74	6767	6443	6454	6443
query75	3402	2703	2660	2660
query76	2880	978	928	928
query77	415	261	264	261
query78	10878	10253	10173	10173
query79	3813	518	513	513
query80	1907	433	430	430
query81	536	222	229	222
query82	1532	100	99	99
query83	322	164	169	164
query84	263	92	84	84
query85	1498	271	314	271
query86	472	294	312	294
query87	3504	3307	3247	3247
query88	4578	2401	2388	2388
query89	474	381	367	367
query90	2012	183	178	178
query91	127	94	99	94
query92	57	47	46	46
query93	4753	506	504	504
query94	1208	179	181	179
query95	389	299	309	299
query96	601	264	265	264
query97	3134	2922	2981	2922
query98	236	218	216	216
query99	1272	873	839	839
Total cold run time: 291977 ms
Total hot run time: 187096 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.19% (8917/25343)
Line Coverage: 26.97% (73332/271897)
Region Coverage: 26.15% (37889/144899)
Branch Coverage: 22.96% (19290/84000)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d50d9cce55188491e6c8516cf23b189b503b2d31_d50d9cce55188491e6c8516cf23b189b503b2d31/report/index.html

Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Apr 27, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 27, 2024
@Lchangliang
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 28, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}
}

TEST_F(BlockFileCacheTest, remove_from_other_queue_2) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]

    ^
Additional context

be/test/io/cache/block_file_cache_test.cpp:3875: 104 lines including whitespace and comments (threshold 80)

    ^

@doris-robot
Copy link

TPC-H: Total hot run time: 41458 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7b94d793c4e736a37bfcc4a2c3c9842f4c1af6c6, data reload: false

------ Round 1 ----------------------------------
q1	17631	4357	4273	4273
q2	2029	192	188	188
q3	10462	1191	1152	1152
q4	10182	835	793	793
q5	7538	2761	2740	2740
q6	229	128	131	128
q7	1069	619	634	619
q8	9216	2199	2162	2162
q9	9188	6827	6863	6827
q10	9000	3967	3968	3967
q11	447	244	241	241
q12	441	225	223	223
q13	18477	3158	3017	3017
q14	287	227	244	227
q15	506	488	495	488
q16	516	421	391	391
q17	987	656	712	656
q18	8403	7865	7836	7836
q19	5301	1587	1564	1564
q20	651	313	326	313
q21	5256	3380	4199	3380
q22	352	273	274	273
Total cold run time: 118168 ms
Total hot run time: 41458 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4494	4416	4463	4416
q2	385	281	265	265
q3	3173	2914	2806	2806
q4	1910	1643	1655	1643
q5	5572	5562	5528	5528
q6	220	122	123	122
q7	2316	1997	1982	1982
q8	3247	3462	3401	3401
q9	8886	8906	8787	8787
q10	4010	3892	3910	3892
q11	588	487	495	487
q12	787	618	646	618
q13	15926	3153	3122	3122
q14	300	297	285	285
q15	512	465	477	465
q16	468	423	422	422
q17	1757	1484	1448	1448
q18	7894	7665	7255	7255
q19	1658	1552	1554	1552
q20	1985	1776	1753	1753
q21	12562	4787	4757	4757
q22	572	493	496	493
Total cold run time: 79222 ms
Total hot run time: 55499 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185932 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7b94d793c4e736a37bfcc4a2c3c9842f4c1af6c6, data reload: false

query1	902	372	354	354
query2	6438	2464	2324	2324
query3	6640	207	208	207
query4	23425	21345	21410	21345
query5	4117	419	412	412
query6	260	188	185	185
query7	4584	279	278	278
query8	240	189	178	178
query9	8460	2293	2274	2274
query10	442	242	252	242
query11	14653	14113	14184	14113
query12	138	86	90	86
query13	1643	376	375	375
query14	8725	8474	7014	7014
query15	240	165	172	165
query16	7935	258	248	248
query17	1747	556	533	533
query18	2028	276	263	263
query19	212	148	146	146
query20	93	85	84	84
query21	193	132	125	125
query22	5058	4922	4945	4922
query23	33999	33185	33127	33127
query24	7405	2946	2983	2946
query25	568	370	376	370
query26	704	148	150	148
query27	2155	318	320	318
query28	5515	2004	2003	2003
query29	838	608	586	586
query30	238	151	151	151
query31	957	729	731	729
query32	89	52	51	51
query33	539	246	235	235
query34	866	468	468	468
query35	754	688	679	679
query36	1089	932	921	921
query37	101	70	68	68
query38	3191	3042	3006	3006
query39	1602	1551	1546	1546
query40	200	127	123	123
query41	42	39	39	39
query42	102	97	96	96
query43	570	537	556	537
query44	1043	728	732	728
query45	272	250	281	250
query46	1078	709	730	709
query47	1954	1861	1859	1859
query48	383	294	289	289
query49	843	418	412	412
query50	781	376	385	376
query51	6709	6670	6596	6596
query52	100	88	89	88
query53	357	275	274	274
query54	259	226	235	226
query55	79	72	69	69
query56	239	221	220	220
query57	1207	1125	1133	1125
query58	219	199	193	193
query59	3563	3223	3148	3148
query60	251	233	232	232
query61	108	89	93	89
query62	588	442	440	440
query63	302	274	277	274
query64	8180	7139	7136	7136
query65	3101	3033	3034	3033
query66	792	343	374	343
query67	15576	15143	15216	15143
query68	5265	530	543	530
query69	484	305	306	305
query70	1200	1118	1089	1089
query71	358	267	284	267
query72	8113	2664	2437	2437
query73	712	313	316	313
query74	6504	6168	6124	6124
query75	3314	2673	2678	2673
query76	2882	944	976	944
query77	385	260	259	259
query78	11026	10404	10263	10263
query79	3585	517	513	513
query80	2137	421	426	421
query81	542	224	221	221
query82	963	100	95	95
query83	274	203	170	170
query84	264	97	85	85
query85	2073	270	258	258
query86	531	313	311	311
query87	3292	3158	3104	3104
query88	4584	2336	2340	2336
query89	485	377	361	361
query90	2047	178	181	178
query91	125	94	94	94
query92	61	53	48	48
query93	5334	517	495	495
query94	1266	182	178	178
query95	389	304	296	296
query96	608	271	265	265
query97	3114	2967	3000	2967
query98	233	230	215	215
query99	1210	867	845	845
Total cold run time: 281041 ms
Total hot run time: 185932 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.51% (8925/25131)
Line Coverage: 27.15% (73482/270660)
Region Coverage: 26.34% (37955/144122)
Branch Coverage: 23.13% (19336/83602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7b94d793c4e736a37bfcc4a2c3c9842f4c1af6c6_7b94d793c4e736a37bfcc4a2c3c9842f4c1af6c6/report/index.html

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit e206e33 into apache:master Apr 30, 2024
24 of 27 checks passed
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 14, 2024
M1saka2003 pushed a commit to M1saka2003/doris that referenced this pull request May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants