Skip to content

[fix](memtable) using memtable memory instead of load memory to calculate adaptive write buffer size#62104

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:fix_adaptive_memtable_size
Apr 8, 2026
Merged

[fix](memtable) using memtable memory instead of load memory to calculate adaptive write buffer size#62104
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:fix_adaptive_memtable_size

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented Apr 3, 2026

Problem

_adaptive_write_buffer_size() used MemoryProfile::load_current_usage() to evaluate memory pressure, but this metric does not include memtable memory. As a result, even when memtable memory is high, the pressure check always reads near zero, causing the write buffer to always use the maximum 4× factor and never scale down — defeating the purpose of adaptive sizing.

Solution

Replace MemoryProfile::load_current_usage() with memtable_memory_limiter()->mem_tracker()->consumption(), which directly reflects actual memtable memory consumption. The scaling logic remains the same:

Memtable memory usage Buffer size factor
> 80% of limit 1× (minimum)
> 50% of limit
≤ 50% of limit 4× (maximum)

A null check is added for safety in case the limiter or tracker is not initialized.

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented Apr 3, 2026

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 28957 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ea570b45ae54bdd8ea4585189eecc4fd216525e, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17628	3655	3675	3655
q2	q3	10688	868	605	605
q4	4683	466	361	361
q5	7461	1341	1142	1142
q6	185	171	144	144
q7	923	962	748	748
q8	9316	1427	1287	1287
q9	5530	5290	5278	5278
q10	6245	2031	1780	1780
q11	473	277	285	277
q12	636	420	312	312
q13	18031	2795	2145	2145
q14	278	281	263	263
q15	q16	860	870	786	786
q17	1078	1137	696	696
q18	6476	5668	5582	5582
q19	1207	1235	1133	1133
q20	565	422	286	286
q21	3962	2727	2088	2088
q22	487	430	389	389
Total cold run time: 96712 ms
Total hot run time: 28957 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4802	4498	4670	4498
q2	q3	4605	4736	4161	4161
q4	2018	2088	1346	1346
q5	4872	4976	5192	4976
q6	229	194	158	158
q7	2213	1894	1607	1607
q8	3288	3150	3106	3106
q9	8308	8159	8233	8159
q10	4475	4444	4200	4200
q11	572	410	392	392
q12	675	734	497	497
q13	2801	3051	2407	2407
q14	307	313	271	271
q15	q16	743	785	690	690
q17	1319	1375	1314	1314
q18	7837	7229	7030	7030
q19	1112	1146	1106	1106
q20	2267	2266	2002	2002
q21	6106	5416	4774	4774
q22	548	496	417	417
Total cold run time: 59097 ms
Total hot run time: 53111 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 178826 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ea570b45ae54bdd8ea4585189eecc4fd216525e, data reload: false

query5	4344	686	530	530
query6	335	240	211	211
query7	4348	596	341	341
query8	346	257	235	235
query9	8733	3936	3941	3936
query10	461	394	331	331
query11	6633	5481	5110	5110
query12	187	133	122	122
query13	1298	657	498	498
query14	5711	5194	4808	4808
query14_1	4128	4158	4130	4130
query15	228	205	181	181
query16	991	459	449	449
query17	1029	774	664	664
query18	2454	505	391	391
query19	231	216	181	181
query20	143	134	132	132
query21	228	146	124	124
query22	14013	14935	14587	14587
query23	18205	16915	16795	16795
query23_1	16713	16706	16636	16636
query24	7408	1725	1344	1344
query24_1	1377	1372	1369	1369
query25	585	521	436	436
query26	1258	332	186	186
query27	2666	664	377	377
query28	4438	1867	1856	1856
query29	946	690	559	559
query30	302	229	191	191
query31	1088	1030	936	936
query32	84	71	74	71
query33	516	353	333	333
query34	1223	1197	665	665
query35	746	790	647	647
query36	1209	1276	1114	1114
query37	154	105	87	87
query38	3103	3001	2954	2954
query39	914	887	865	865
query39_1	817	847	834	834
query40	242	150	128	128
query41	63	59	63	59
query42	110	113	106	106
query43	320	314	281	281
query44	
query45	202	197	182	182
query46	1122	1263	815	815
query47	2314	2319	2236	2236
query48	403	401	301	301
query49	637	541	413	413
query50	777	298	241	241
query51	4336	4250	4218	4218
query52	110	107	97	97
query53	244	275	199	199
query54	327	279	264	264
query55	101	97	89	89
query56	298	312	315	312
query57	1656	1754	1632	1632
query58	341	274	275	274
query59	2948	2976	2758	2758
query60	342	334	328	328
query61	157	155	146	146
query62	669	617	576	576
query63	253	194	195	194
query64	5311	1305	998	998
query65	
query66	1487	483	362	362
query67	24245	24221	24203	24203
query68	
query69	439	339	316	316
query70	1074	989	1003	989
query71	327	280	280	280
query72	2957	2705	2425	2425
query73	781	810	431	431
query74	9933	9805	9541	9541
query75	2753	2633	2316	2316
query76	2298	1142	798	798
query77	421	424	346	346
query78	11277	11331	10757	10757
query79	1494	1145	809	809
query80	737	635	536	536
query81	450	280	242	242
query82	1389	173	130	130
query83	379	304	268	268
query84	290	164	133	133
query85	939	491	440	440
query86	396	313	332	313
query87	3336	3197	3069	3069
query88	3645	2702	2676	2676
query89	444	395	362	362
query90	1980	179	178	178
query91	183	184	138	138
query92	77	76	70	70
query93	983	972	558	558
query94	529	364	304	304
query95	652	458	341	341
query96	1097	781	336	336
query97	2676	2653	2555	2555
query98	244	234	233	233
query99	1065	1068	953	953
Total cold run time: 257351 ms
Total hot run time: 178826 ms

@doris-robot
Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.01% (20085/37892)
Line Coverage 36.56% (188610/515830)
Region Coverage 32.84% (146568/446288)
Branch Coverage 33.97% (64130/188790)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.64% (27326/37108)
Line Coverage 57.20% (294135/514249)
Region Coverage 54.48% (245397/450422)
Branch Coverage 56.19% (106411/189372)

Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 8, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 8, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 8, 2026

PR approved by anyone and no changes requested.

Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 7ad6489 into apache:master Apr 8, 2026
33 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants