[improvement](fe) Improve external catalog meta cache observability#63809
Open
suxiaogang223 wants to merge 2 commits into
Open
[improvement](fe) Improve external catalog meta cache observability#63809suxiaogang223 wants to merge 2 commits into
suxiaogang223 wants to merge 2 commits into
Conversation
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The catalog_meta_cache_statistics table exposed cumulative eviction count for external catalog meta cache entries, but did not provide a direct replacement frequency metric. Users had to derive replacement pressure manually when checking whether the configured meta cache capacity was too small. This change adds EVICTION_RATE, computed as eviction_count / request_count with zero returned when there are no requests, and exposes it beside EVICTION_COUNT in the information schema result.
### Release note
Add EVICTION_RATE to information_schema.catalog_meta_cache_statistics for observing catalog meta cache replacement frequency.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTest
- Behavior changed: Yes (catalog_meta_cache_statistics now includes EVICTION_RATE)
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: The default Hive partition metadata cache capacity was too small for common external catalog workloads, which could cause frequent evictions even without explicit capacity tuning. This change raises the single Hive partition metadata cache default from 10,000 to 100,000 and raises the Hive partitioned-table values cache default from 1,000 to 10,000. External fuzzy testing now includes the new default-size candidates. While checking similar cache entries, MaxCompute partition_values was found to cache table-level partition value structures but reused the single-partition Hive capacity; it now follows the table-level partition values capacity instead.
### Release note
Increase default Hive partition meta cache capacities to reduce frequent evictions. MaxCompute partition_values cache now uses the table-level partition values capacity setting.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.datasource.hive.HiveMetaStoreCacheTest,org.apache.doris.datasource.maxcompute.MaxComputeExternalMetaCacheTest
- Behavior changed: Yes (default Hive partition meta cache capacities are larger, and MaxCompute partition_values uses the table-level partition values capacity)
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31577 ms |
Contributor
TPC-DS: Total hot run time: 172185 ms |
Gabriel39
approved these changes
May 28, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review opinion: no blocking issues found in the PR diff.
Critical checkpoint conclusions:
- Goal/test: The PR adds EVICTION_RATE, increases Hive partition metadata cache defaults, and adjusts MaxCompute partition_values capacity. The changed code and added unit tests cover these goals.
- Scope: The changes are small and focused on cache configuration and information_schema metadata exposure.
- Concurrency/lifecycle: No new concurrency or special lifecycle risk found; cache specs are still captured during catalog cache initialization as before.
- Configuration: Existing config defaults are changed, with fuzzy config updated accordingly. No new dynamic config behavior is introduced.
- Compatibility: The information_schema table gains a new trailing column; row generation was updated in the same order.
- Parallel paths: Hive and MaxCompute partition-values cache paths were checked; the MaxCompute path now uses the intended table-level capacity.
- Tests: I attempted to run ./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTest, but the runner lacks thirdparty/installed/bin/protoc, so generated-source setup failed before the test executed. I did not run the remaining FE unit tests for the same environment reason.
- Observability: The new EVICTION_RATE is exposed through catalog_meta_cache_statistics and computed safely as 0 when there are no requests.
User focus: No additional user-provided review focus was present.
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
External catalog meta cache statistics exposed cumulative eviction count, but did not provide a direct replacement frequency metric for judging whether cache capacity is too small. This PR adds
EVICTION_RATEtoinformation_schema.catalog_meta_cache_statistics, calculated aseviction_count / request_countand returned as0when there are no requests.Hive partition metadata cache defaults were also too small for common external catalog workloads, causing frequent evictions without explicit tuning. This PR increases the default Hive single-partition cache capacity from 10,000 to 100,000 and the Hive partitioned-table values cache capacity from 1,000 to 10,000. While checking similar cache entries, MaxCompute
partition_valueswas found to cache table-level partition value structures but use the Hive single-partition capacity; it now follows the table-level partition values capacity.Release note
Add
EVICTION_RATEtoinformation_schema.catalog_meta_cache_statistics, increase default Hive partition meta cache capacities, and make MaxComputepartition_valuesuse the table-level partition values capacity.Check List (For Author)
Test
./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTest./run-fe-ut.sh --run org.apache.doris.datasource.hive.HiveMetaStoreCacheTest,org.apache.doris.datasource.maxcompute.MaxComputeExternalMetaCacheTestBehavior changed:
catalog_meta_cache_statisticsincludesEVICTION_RATE; default Hive partition meta cache capacities are larger; MaxComputepartition_valuesuses the table-level partition values capacity.Does this need documentation?
Check List (For Reviewer who merge this PR)