branch-4.0: [fix](cloud) Fix warm up cache not supporting packed file (#60375)#61887
Open
liaoxin01 wants to merge 1 commit intoapache:branch-4.0from
Open
branch-4.0: [fix](cloud) Fix warm up cache not supporting packed file (#60375)#61887liaoxin01 wants to merge 1 commit intoapache:branch-4.0from
liaoxin01 wants to merge 1 commit intoapache:branch-4.0from
Conversation
When enable_packed_file is enabled, the warm up functions fail to download cache data because they use storage_resource->fs (the raw FileSystem) instead of RowsetMeta::fs() which wraps the FileSystem with PackedFileSystem. The PackedFileSystem correctly maps segment file paths to their actual locations within packed files. Without this mapping, the download fails because the segment files don't exist on S3 - they are packed into larger packed files. This fix changes the file_system used in: 1. download_file_cache_block (block_file_cache_downloader.cpp) 2. _submit_segment_download_task (cloud_tablet.cpp) 3. _submit_inverted_index_download_task (cloud_tablet.cpp) All now use RowsetMeta::fs() instead of storage_resource->fs to properly handle packed files.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes cloud warm-up file cache downloads to correctly support packed files by using the PackedFileSystem-wrapped filesystem from rowset metadata, and extends regression coverage to run warm-up/cache/tablet-cleanup suites with packed files toggled.
Changes:
- Switch warm-up download paths to use
RowsetMeta::fs()(PackedFileSystem-aware) instead ofstorage_resource->fs. - Update multiple cloud P0 regression suites to run with
enable_packed_fileenabled/disabled. - Add BE config injection for
enable_packed_fileacross the impacted suites.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 18 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_rebalance.groovy | Randomly toggles enable_packed_file and passes it into BE configs for rebalance cleanup coverage. |
| regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_drop_force_table.groovy | Randomly toggles enable_packed_file and passes it into BE configs for drop-force cleanup coverage. |
| regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_index_file_cache.groovy | Randomly toggles enable_packed_file and passes it into BE configs for stale index cache cleanup coverage. |
| regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_file_cache.groovy | Randomly toggles enable_packed_file and passes it into BE configs for stale segment cache cleanup coverage. |
| regression-test/suites/cloud_p0/cache/test_topn_broadcast.groovy | Randomly toggles enable_packed_file and passes it into BE configs for cache behavior coverage. |
| regression-test/suites/cloud_p0/cache/test_load_cache.groovy | Randomly toggles enable_packed_file and passes it into BE configs for load/query cache behavior coverage. |
| regression-test/suites/cloud_p0/balance/test_warmup_rebalance.groovy | Randomly toggles enable_packed_file and applies it to BE configs for rebalance warm-up coverage. |
| regression-test/suites/cloud_p0/balance/test_peer_read_async_warmup.groovy | Randomly toggles enable_packed_file and applies it to BE configs for peer-read warm-up coverage. |
| regression-test/suites/cloud_p0/balance/test_expanding_node_balance.groovy | Randomly toggles enable_packed_file and applies it to BE configs across multiple clusters. |
| regression-test/suites/cloud_p0/balance/test_balance_warm_up_with_compaction_use_peer_cache.groovy | Randomly toggles enable_packed_file and applies it to BE configs for warm-up + compaction scenario. |
| regression-test/suites/cloud_p0/balance/test_balance_warm_up_use_peer_cache.groovy | Randomly toggles enable_packed_file and applies it to BE configs for warm-up using peer cache. |
| regression-test/suites/cloud_p0/balance/test_balance_warm_up_task_abnormal.groovy | Randomly toggles enable_packed_file and applies it to BE configs for abnormal warm-up task behavior. |
| regression-test/suites/cloud_p0/balance/test_balance_warm_up_sync_global_config.groovy | Randomly toggles enable_packed_file and applies it to BE configs for sync warm-up config path. |
| regression-test/suites/cloud_p0/balance/test_balance_warm_up.groovy | Randomly toggles enable_packed_file and applies it to BE configs for baseline warm-up balance coverage. |
| regression-test/suites/cloud_p0/balance/test_balance_use_compute_group_properties.groovy | Randomly toggles enable_packed_file and applies it to BE configs for compute group property interactions. |
| regression-test/suites/cloud_p0/balance/test_balance_metrics.groovy | Randomly toggles enable_packed_file and applies it to BE configs for balance metrics assertions. |
| regression-test/suites/cloud_p0/balance/test_alter_compute_group_properties.groovy | Randomly toggles enable_packed_file and applies it to BE configs for compute group alter flows. |
| be/src/io/cache/block_file_cache_downloader.cpp | Uses RowsetMeta::fs() to ensure warm-up downloads resolve packed segment paths correctly. |
| be/src/cloud/cloud_tablet.cpp | Uses rowset_meta->fs() when submitting warm-up download tasks so packed files are handled correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_drop_force_table.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/balance/test_balance_use_compute_group_properties.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_rebalance.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/balance/test_balance_warm_up_task_abnormal.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_file_cache.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/balance/test_peer_read_async_warmup.groovy
Show resolved
Hide resolved
...sion-test/suites/cloud_p0/balance/test_balance_warm_up_with_compaction_use_peer_cache.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/balance/test_balance_warm_up_use_peer_cache.groovy
Show resolved
Hide resolved
regression-test/suites/cloud_p0/balance/test_alter_compute_group_properties.groovy
Show resolved
Hide resolved
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
liaoxin01
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pick #60375