[feature](fe) Add byte-weighted metadata cache framework by CalvinKirs · Pull Request #66533 · apache/doris

CalvinKirs · 2026-08-06T07:03:17Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

External metadata caches are currently bounded only by entry count, so entries with very different memory footprints are treated equally. This PR adds framework support for an optional catalog-level max-weight, an entry-specific size estimator contract, mutually exclusive Caffeine size/weight construction, saturated integer weight conversion, and weighted cache statistics.

It also completes the framework by supporting weighted caches with synchronous removal listeners, accepting binary size suffixes such as 512MB while retaining bare-byte compatibility, strictly rejecting malformed, negative, and overflowing values, and keeping statistics reads lightweight and side-effect free.

Existing entries continue to use maximumSize unless they explicitly register an estimator and configure max-weight. Catalog-specific Iceberg and Paimon estimators and information_schema exposure are intentionally not included.

What the framework does

For a cache entry that opts in, the framework asks its MetaCacheSizeEstimator to estimate the bytes retained by each key/value pair and passes that value to Caffeine's Weigher. Caffeine then bounds the cache with maximumWeight instead of maximumSize. Oversized estimates are saturated to Caffeine's integer weight limit, negative estimates are rejected, and weight-based eviction also works with synchronous removal listeners.

An entry opts in explicitly when its definition registers an estimator:

MetaCacheEntryDef.of(...)
        .withSizeEstimator((key, value) -> estimatedBytes);

This keeps estimation entry-specific: the owner of each metadata type decides what its key/value pair retains. This PR does not add a generic JVM object-size utility or automatically enable weighted eviction for existing entries.

Future catalog configuration

After an engine entry registers an estimator, a catalog can select weighted eviction with the standard entry property:

meta.cache.<engine>.<entry>.enable=true
meta.cache.<engine>.<entry>.ttl-second=3600
meta.cache.<engine>.<entry>.max-weight=512MB

max-weight accepts binary B, KB, MB, GB, TB, and PB suffixes (case-insensitive); a bare integer remains bytes for compatibility. For example, 512MB is parsed as 512 * 1024 * 1024 bytes. Malformed, negative, decimal, unknown-unit, and overflowing values are rejected. A configured max-weight without an entry estimator is also rejected.

Count-bounded mode remains unchanged: omit max-weight and configure capacity as before.

Configuration relationships

Setting	Relationship and effect
`enable`	Master switch. `false` disables the entry cache.
`ttl-second`	Independent time policy. `0` disables the cache, `-1` means no expiration, and a positive value enables expire-after-access.
`max-weight` vs. `capacity`	Alternative bounds, not additive. If `max-weight` is present, weighted mode is selected and `capacity` is ignored; otherwise count-based `capacity` is used.
Selected bound value	A selected bound of `0` disables the cache.
TTL vs. selected bound	Both policies can be active together. At runtime an entry may be removed when either the TTL expires or the selected count/weight bound requires eviction.

In short, effective enablement is enable && ttl-second != 0 && selected-bound != 0, where selected-bound is max-weight when present and capacity otherwise.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
  - ./run-fe-ut.sh --run org.apache.doris.common.util.ParseUtilTest,org.apache.doris.common.CacheFactoryTest,org.apache.doris.datasource.metacache.CacheSpecTest,org.apache.doris.datasource.metacache.MetaCacheEntryTest,org.apache.doris.datasource.metacache.AbstractExternalMetaCacheTest (80 tests passed)
  - DISABLE_BUILD_UI=ON ./build.sh --fe (passed, including Checkstyle)
- Manual test
- No need to test or manual test
Behavior changed:
- No.
- Yes. Entries that register an estimator and configure max-weight use weighted eviction, invalid values are rejected, and statistics reads no longer trigger Caffeine maintenance.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-06T07:03:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: External metadata caches are currently bounded only by entry count. Add framework support for an optional catalog-level max-weight, an entry-specific size estimator contract, mutually exclusive Caffeine size/weight construction, saturated integer weight conversion, and weighted cache statistics. Complete the framework by supporting weighted caches with synchronous removal listeners, accepting convenient binary size suffixes such as MB while retaining bare-byte compatibility, strictly rejecting malformed, negative, and overflowing max-weight values, and keeping statistics reads lightweight and side-effect free. Existing entries continue to use maximumSize unless they explicitly register an estimator and configure max-weight. Catalog-specific estimators and information_schema exposure are intentionally not included. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.common.util.ParseUtilTest,org.apache.doris.common.CacheFactoryTest,org.apache.doris.datasource.metacache.CacheSpecTest,org.apache.doris.datasource.metacache.MetaCacheEntryTest,org.apache.doris.datasource.metacache.AbstractExternalMetaCacheTest (80 tests passed) - DISABLE_BUILD_UI=ON ./build.sh --fe (passed, including Checkstyle) - Behavior changed: Yes. Entries that register an estimator and configure max-weight use weighted eviction, invalid max-weight values are rejected, and statistics reads no longer trigger Caffeine maintenance. - Does this need documentation: No

CalvinKirs · 2026-08-06T09:00:36Z

run buildall

hello-stephen · 2026-08-06T10:40:37Z

FE UT Coverage Report

Increment line coverage 97.30% (108/111) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-06T12:22:53Z

TPC-H: Total hot run time: 29083 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4061bdee4f9633517b83d19d43908081f33ee1f1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17699	3941	3921	3921
q2	1989	323	199	199
q3	10307	1402	788	788
q4	4682	468	340	340
q5	7531	831	565	565
q6	187	176	135	135
q7	747	818	587	587
q8	9318	1510	1532	1510
q9	5294	4040	4067	4040
q10	6736	1638	1356	1356
q11	500	356	333	333
q12	733	586	450	450
q13	18055	3222	2729	2729
q14	260	262	235	235
q15	q16	735	735	672	672
q17	1047	1110	876	876
q18	6563	5637	5554	5554
q19	1175	1290	1121	1121
q20	794	724	587	587
q21	5734	2818	2767	2767
q22	473	373	318	318
Total cold run time: 100559 ms
Total hot run time: 29083 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4951	4708	4609	4609
q2	290	321	217	217
q3	4861	5236	4657	4657
q4	2153	2311	1423	1423
q5	4554	4619	4457	4457
q6	230	174	122	122
q7	1821	1817	1500	1500
q8	2339	2016	2018	2016
q9	7218	7039	6710	6710
q10	4218	4188	3795	3795
q11	511	372	333	333
q12	684	700	496	496
q13	2934	3197	2771	2771
q14	266	272	258	258
q15	q16	659	680	588	588
q17	1247	1232	1238	1232
q18	12151	10995	11932	10995
q19	1091	1065	1106	1065
q20	2191	2201	1909	1909
q21	5257	4492	4554	4492
q22	524	442	412	412
Total cold run time: 60150 ms
Total hot run time: 54057 ms

hello-stephen · 2026-08-06T12:24:26Z

FE Regression Coverage Report

Increment line coverage 30.63% (34/111) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-06T12:33:29Z

TPC-DS: Total hot run time: 166250 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4061bdee4f9633517b83d19d43908081f33ee1f1, data reload: false

query5	4323	579	433	433
query6	454	215	198	198
query7	4888	532	336	336
query8	323	163	145	145
query9	8767	4036	4025	4025
query10	460	371	306	306
query11	5847	2206	2035	2035
query12	156	112	97	97
query13	1267	629	444	444
query14	6067	4288	3965	3965
query14_1	3805	3797	3764	3764
query15	198	190	174	174
query16	971	469	431	431
query17	891	681	515	515
query18	2413	467	330	330
query19	205	183	136	136
query20	102	98	99	98
query21	227	155	137	137
query22	13029	12976	12832	12832
query23	15695	15121	14495	14495
query23_1	14738	14767	14593	14593
query24	7617	1697	1214	1214
query24_1	1239	1247	1261	1247
query25	521	414	351	351
query26	1331	346	209	209
query27	2594	608	383	383
query28	4572	2075	2024	2024
query29	1054	587	468	468
query30	344	268	229	229
query31	1164	1117	1035	1035
query32	110	61	58	58
query33	502	296	261	261
query34	1174	1111	656	656
query35	723	738	637	637
query36	779	778	693	693
query37	153	107	92	92
query38	1825	1755	1685	1685
query39	822	818	806	806
query39_1	791	784	786	784
query40	252	157	141	141
query41	70	63	63	63
query42	92	100	94	94
query43	318	319	282	282
query44	1386	759	779	759
query45	192	178	171	171
query46	1025	1133	714	714
query47	1581	1540	1465	1465
query48	395	439	318	318
query49	592	402	308	308
query50	1057	419	342	342
query51	10455	10590	10619	10590
query52	85	88	76	76
query53	260	279	204	204
query54	297	248	239	239
query55	78	72	68	68
query56	328	302	292	292
query57	1055	1022	980	980
query58	288	288	262	262
query59	1531	1614	1368	1368
query60	333	284	258	258
query61	175	174	171	171
query62	399	326	263	263
query63	237	201	197	197
query64	2978	1139	962	962
query65	3900	3805	3827	3805
query66	1855	498	373	373
query67	28212	28098	27992	27992
query68	3338	1610	1017	1017
query69	419	315	312	312
query70	889	767	776	767
query71	379	359	308	308
query72	3006	2650	2340	2340
query73	849	782	423	423
query74	4604	4495	4285	4285
query75	2375	2327	1979	1979
query76	2418	1127	787	787
query77	329	353	280	280
query78	11039	11146	10602	10602
query79	1353	1119	724	724
query80	736	541	471	471
query81	468	330	301	301
query82	617	168	133	133
query83	396	327	291	291
query84	329	154	131	131
query85	955	620	527	527
query86	346	237	229	229
query87	1972	1944	1847	1847
query88	3681	2780	2797	2780
query89	397	316	278	278
query90	1843	198	187	187
query91	203	191	162	162
query92	64	61	54	54
query93	1615	1535	988	988
query94	617	365	322	322
query95	784	534	480	480
query96	1081	795	368	368
query97	2470	2471	2386	2386
query98	202	190	184	184
query99	747	728	608	608
Total cold run time: 252136 ms
Total hot run time: 166250 ms

hello-stephen · 2026-08-06T12:38:29Z

ClickBench: Total hot run time: 24.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4061bdee4f9633517b83d19d43908081f33ee1f1, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.04
query3	0.26	0.14	0.14
query4	1.62	0.15	0.14
query5	0.25	0.23	0.23
query6	1.15	0.85	0.82
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.33	0.33
query10	0.59	0.58	0.64
query11	0.19	0.15	0.14
query12	0.18	0.14	0.15
query13	0.49	0.49	0.51
query14	1.05	1.03	1.03
query15	0.65	0.61	0.61
query16	0.34	0.32	0.34
query17	1.16	1.15	1.08
query18	0.23	0.22	0.22
query19	2.03	2.03	1.97
query20	0.01	0.02	0.01
query21	15.43	0.22	0.14
query22	4.77	0.06	0.05
query23	16.12	0.31	0.12
query24	2.95	0.43	0.34
query25	0.10	0.05	0.05
query26	0.72	0.20	0.14
query27	0.03	0.03	0.04
query28	3.60	0.82	0.38
query29	12.51	4.24	3.29
query30	0.27	0.15	0.16
query31	2.78	0.59	0.34
query32	3.23	0.60	0.51
query33	3.20	3.26	3.18
query34	15.48	4.24	3.46
query35	3.42	3.45	3.44
query36	0.58	0.46	0.43
query37	0.08	0.06	0.07
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 96.5 s
Total hot run time: 24.76 s

### What problem does this PR solve? Issue Number: None Related PR: apache#66533 Problem Summary: Add opt-in JMH benchmarks for the production Hive file-listing and Iceberg partition and manifest cache estimators. The benchmarks compare the constant-time cached weight lookup, the one-time value construction and estimation cost, and JOL retained-graph traversal without adding benchmark dependencies to the default FE reactor. ### Release note None ### Check List (For Author) - Test: Manual test - `mvn -Pbenchmark -pl fe-benchmark -am test-compile -DskipTests` - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#66533 Problem Summary: Paimon's derived partition-view cache was limited only by entry count, so a catalog containing large partition views could retain substantially more FE heap than its configured entry capacity implied. Extend the generic connector metadata cache to accept a type-specific estimator, add a Paimon partition-view estimator that computes the complete immutable entry weight once when max-weight is enabled, and use the stored value for O(1) Caffeine weighing. Keep the legacy count-bounded path unchanged. Add JMH/JOL benchmarks for 10,000 and 100,000 Paimon partitions. ### Release note Paimon partition-view caches support the catalog property `meta.cache.paimon.partition_view.max-weight`. ### Check List (For Author) - Test: Unit Test - `./run-fe-ut.sh --run org.apache.doris.connector.cache.ConnectorMetadataCacheTest,org.apache.doris.connector.paimon.PaimonPartitionViewSizeEstimatorTest,org.apache.doris.connector.paimon.PaimonConnectorValidatePropertiesTest` - `mvn -Pbenchmark -pl fe-benchmark -am test-compile -DskipTests` - Behavior changed: Yes. Paimon partition-view caches can opt into byte-weighted eviction; existing capacity behavior remains the default. - Does this need documentation: No

CalvinKirs marked this pull request as ready for review August 6, 2026 07:05

CalvinKirs force-pushed the master-catalog-cache-memory branch 2 times, most recently from 7c1f1e2 to 710cdb8 Compare August 6, 2026 08:20

CalvinKirs force-pushed the master-catalog-cache-memory branch from 710cdb8 to 4061bde Compare August 6, 2026 08:50

CalvinKirs mentioned this pull request Aug 7, 2026

[feature](fe) Add byte-weighted external metadata caches #66579

Draft

16 tasks

Navigation Menu

Uh oh!

[feature](fe) Add byte-weighted metadata cache framework - #66533

[feature](fe) Add byte-weighted metadata cache framework#66533
CalvinKirs wants to merge 1 commit into
apache:masterfrom
CalvinKirs:master-catalog-cache-memory

CalvinKirs commented Aug 6, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

CalvinKirs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

CalvinKirs commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What the framework does

Future catalog configuration

Configuration relationships

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

CalvinKirs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CalvinKirs commented Aug 6, 2026 •

edited

Loading