Skip to content

branch-3.1: [fix](filecache) handle duplicated block when enable lru persist #53648#55401

Merged
morrySnow merged 1 commit intoapache:branch-3.1from
freemandealer:pick-lru-persist-fix
Aug 28, 2025
Merged

branch-3.1: [fix](filecache) handle duplicated block when enable lru persist #53648#55401
morrySnow merged 1 commit intoapache:branch-3.1from
freemandealer:pick-lru-persist-fix

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Aug 28, 2025 •
edited by morrySnow
Loading

picked from #53648

@freemandealer
[fix](filecache) handle duplicated block when enable lru persist
d8b956b 
Duplicate elements may occur during the transition between TTL and normal states.
- Replay and dump operations are periodic, which means there could be a time window
  that leads to capturing the addition of an element to a queue but not its removal
  from the previous queue.
- The issue stems from the use of non-order-assure current queue in the log queue
  for record operations.

The above two are currently unavoidable, so a fallback logic is necessary to remove duplicates when elements are added repeatedly.

Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

freemandealer commented Aug 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-H: Total hot run time: 32892 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8b956b322021faa91124c289a21fe266a7aefa1, data reload: false

------ Round 1 ----------------------------------
q1	17630	5552	5417	5417
q2	2024	385	281	281
q3	11663	1255	780	780
q4	10576	878	458	458
q5	9610	2416	2179	2179
q6	196	166	136	136
q7	909	746	614	614
q8	9324	1434	1199	1199
q9	5296	4923	4941	4923
q10	6742	2294	1855	1855
q11	472	290	270	270
q12	341	357	209	209
q13	17800	3626	3001	3001
q14	240	234	214	214
q15	533	478	460	460
q16	425	437	385	385
q17	589	874	360	360
q18	6941	6397	6491	6397
q19	1205	948	536	536
q20	332	351	215	215
q21	2845	2249	2012	2012
q22	1069	1004	991	991
Total cold run time: 106762 ms
Total hot run time: 32892 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5481	5475	5481	5475
q2	239	321	245	245
q3	2229	2632	2314	2314
q4	1345	1776	1364	1364
q5	4413	5025	4978	4978
q6	172	163	128	128
q7	2125	1987	1796	1796
q8	2702	2862	2720	2720
q9	7323	7244	7286	7244
q10	3035	3232	2824	2824
q11	602	504	488	488
q12	671	784	591	591
q13	3439	3794	3134	3134
q14	276	317	281	281
q15	514	474	469	469
q16	441	480	436	436
q17	1236	1761	1275	1275
q18	7664	7453	7379	7379
q19	830	1165	1078	1078
q20	2029	2060	1927	1927
q21	5387	4868	4488	4488
q22	1115	1107	1048	1048
Total cold run time: 53268 ms
Total hot run time: 51682 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

TPC-DS: Total hot run time: 192762 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8b956b322021faa91124c289a21fe266a7aefa1, data reload: false

query1	955	403	423	403
query2	6147	2035	2010	2010
query3	8685	200	199	199
query4	33573	23866	23432	23432
query5	3560	607	464	464
query6	295	184	165	165
query7	4198	481	305	305
query8	294	241	233	233
query9	9452	2623	2578	2578
query10	473	323	265	265
query11	17862	15389	15092	15092
query12	169	108	109	108
query13	1552	533	416	416
query14	9908	7149	6808	6808
query15	251	194	182	182
query16	8114	610	521	521
query17	1544	764	609	609
query18	2153	411	334	334
query19	233	195	176	176
query20	125	125	119	119
query21	207	133	108	108
query22	4476	4562	4401	4401
query23	35140	34397	34386	34386
query24	7585	2766	2756	2756
query25	547	503	445	445
query26	1160	289	193	193
query27	2131	480	361	361
query28	5403	2229	2188	2188
query29	802	615	459	459
query30	253	193	162	162
query31	1013	922	826	826
query32	95	59	61	59
query33	506	379	327	327
query34	742	873	536	536
query35	820	799	749	749
query36	1025	1082	940	940
query37	100	89	68	68
query38	4148	4059	4009	4009
query39	1533	1496	1533	1496
query40	213	114	109	109
query41	61	48	50	48
query42	128	111	110	110
query43	546	542	507	507
query44	1367	846	848	846
query45	185	183	183	183
query46	879	1048	683	683
query47	1991	1963	1959	1959
query48	416	445	343	343
query49	783	515	418	418
query50	670	699	437	437
query51	7380	7357	7142	7142
query52	104	104	98	98
query53	235	258	203	203
query54	551	564	478	478
query55	79	75	80	75
query56	275	270	254	254
query57	1288	1303	1225	1225
query58	222	217	228	217
query59	3279	3389	3209	3209
query60	282	284	274	274
query61	114	108	110	108
query62	807	763	709	709
query63	240	198	193	193
query64	4492	1026	647	647
query65	3365	3304	3383	3304
query66	978	425	310	310
query67	16210	15732	15442	15442
query68	7795	824	531	531
query69	491	304	270	270
query70	1137	1117	1077	1077
query71	390	308	267	267
query72	5696	3758	3700	3700
query73	643	747	355	355
query74	10573	9434	9233	9233
query75	3197	3156	2692	2692
query76	3169	1168	768	768
query77	625	365	300	300
query78	10480	10488	9609	9609
query79	3199	896	598	598
query80	697	529	433	433
query81	523	268	220	220
query82	577	116	86	86
query83	172	159	147	147
query84	255	95	80	80
query85	771	374	299	299
query86	390	287	295	287
query87	4313	4352	4242	4242
query88	5208	2397	2380	2380
query89	400	341	296	296
query90	1771	193	196	193
query91	137	141	109	109
query92	64	64	63	63
query93	2038	900	549	549
query94	662	420	306	306
query95	343	285	278	278
query96	489	614	285	285
query97	3241	3305	3176	3176
query98	229	213	201	201
query99	1532	1408	1313	1313
Total cold run time: 295576 ms
Total hot run time: 192762 ms

@doris-robot
Copy link

doris-robot commented Aug 28, 2025

ClickBench: Total hot run time: 29.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d8b956b322021faa91124c289a21fe266a7aefa1, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.25	0.07	0.06
query4	1.63	0.11	0.10
query5	0.54	0.53	0.51
query6	1.12	0.75	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.49
query10	0.56	0.55	0.56
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	0.78	0.79	0.80
query15	0.84	0.83	0.82
query16	0.39	0.39	0.40
query17	1.06	1.04	1.06
query18	0.23	0.22	0.21
query19	1.95	1.93	1.87
query20	0.02	0.02	0.01
query21	15.36	0.86	0.59
query22	0.75	0.81	0.75
query23	14.99	1.47	0.58
query24	3.45	0.98	1.95
query25	0.19	0.14	0.10
query26	0.31	0.15	0.14
query27	0.05	0.04	0.05
query28	13.50	0.95	0.44
query29	12.54	3.92	3.28
query30	0.24	0.08	0.06
query31	2.85	0.58	0.38
query32	3.23	0.54	0.46
query33	3.02	3.06	3.03
query34	16.41	5.16	4.53
query35	4.63	4.58	4.53
query36	0.65	0.49	0.47
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.64 s
Total hot run time: 29.16 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 28, 2025

BE UT Coverage Report

Increment line coverage 77.78% (7/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.49% (12725/27973)
Line Coverage 36.37% (113450/311908)
Region Coverage 34.01% (64947/190976)
Branch Coverage 31.05% (34099/109814)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.30% (20986/27503)
Line Coverage 69.66% (216546/310851)
Region Coverage 67.68% (129748/191710)
Branch Coverage 61.19% (67499/110306)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.30% (20986/27503)
Line Coverage 69.67% (216567/310851)
Region Coverage 67.69% (129763/191710)
Branch Coverage 61.20% (67508/110306)

@morrySnow morrySnow changed the title [fix](filecache) handle duplicated block when enable lru persist (pick#53648) branch-3.1: [fix](filecache) handle duplicated block when enable lru persist #53648 Aug 28, 2025
@morrySnow morrySnow merged commit 7d1ca2d into apache:branch-3.1 Aug 28, 2025
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@freemandealer @doris-robot @hello-stephen @gavinchou @morrySnow