Skip to content

[fix](cloud) Fix warm up cache not supporting packed file (#60375)#61886

Merged
yiguolei merged 1 commit intoapache:branch-4.1from
liaoxin01:cherry-pick-60375-to-branch-4.1
Mar 31, 2026
Merged

[fix](cloud) Fix warm up cache not supporting packed file (#60375)#61886
yiguolei merged 1 commit intoapache:branch-4.1from
liaoxin01:cherry-pick-60375-to-branch-4.1

Conversation

@liaoxin01
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 commented Mar 30, 2026

Pick #60375

@liaoxin01
[fix](cloud) Fix warm up cache not supporting packed file (apache#60375)
d5d3d3a 
When enable_packed_file is enabled, the warm up functions fail to
download cache data because they use storage_resource->fs (the raw
FileSystem) instead of RowsetMeta::fs() which wraps the FileSystem with
PackedFileSystem.

The PackedFileSystem correctly maps segment file paths to their actual
locations within packed files. Without this mapping, the download fails
because the segment files don't exist on S3 - they are packed into
larger packed files.

This fix changes the file_system used in:
1. download_file_cache_block (block_file_cache_downloader.cpp)
2. _submit_segment_download_task (cloud_tablet.cpp)
3. _submit_inverted_index_download_task (cloud_tablet.cpp)

All now use RowsetMeta::fs() instead of storage_resource->fs to properly
handle packed files.
Copilot AI review requested due to automatic review settings March 30, 2026 08:59
@liaoxin01 liaoxin01 requested a review from yiguolei as a code owner March 30, 2026 08:59
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Mar 30, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Copy Markdown
Contributor Author

liaoxin01 commented Mar 30, 2026

run buildall

Copilot AI reviewed Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes cloud warm-up/file-cache downloads to correctly support packed files by switching download I/O to use the RowsetMeta-provided filesystem wrapper, and updates cloud regression suites to exercise packed-file mode.

Changes:

  • Use RowsetMeta::fs() / rowset_meta->fs() for warm-up and file-cache downloads so segment/index paths resolve correctly when packed files are used.
  • Update multiple cloud regression suites to toggle enable_packed_file at runtime to cover both packed/non-packed scenarios.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 20 comments.

Show a summary per file
File Description
be/src/io/cache/block_file_cache_downloader.cpp Use RowsetMeta::fs() for file-cache block downloads to support packed-file path mapping.
be/src/cloud/cloud_tablet.cpp Use rowset_meta->fs() for warm-up download tasks (segment + inverted index) to support packed files.
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_rebalance.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/tablets/test_clean_tablet_when_drop_force_table.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_index_file_cache.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/tablets/test_clean_stale_rs_file_cache.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/cache/test_topn_broadcast.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/cache/test_load_cache.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_warmup_rebalance.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_peer_read_async_warmup.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_expanding_node_balance.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_with_compaction_use_peer_cache.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_use_peer_cache.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_task_abnormal.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_warm_up_sync_global_config.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_warm_up.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_use_compute_group_properties.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_balance_metrics.groovy Randomize enable_packed_file during test to cover packed/non-packed.
regression-test/suites/cloud_p0/balance/test_alter_compute_group_properties.groovy Randomize enable_packed_file during test to cover packed/non-packed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Mar 31, 2026

BE UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.84% (19674/37235)
Line Coverage 36.22% (184136/508342)
Region Coverage 32.52% (142479/438134)
Branch Coverage 33.70% (62596/185738)

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Mar 31, 2026

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.74% (26514/36449)
Line Coverage 56.21% (284793/506638)
Region Coverage 53.82% (237959/442102)
Branch Coverage 55.50% (103377/186266)

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Mar 31, 2026

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.74% (26514/36449)
Line Coverage 56.21% (284776/506638)
Region Coverage 53.81% (237895/442102)
Branch Coverage 55.49% (103362/186266)

@yiguolei yiguolei merged commit 61b30b0 into apache:branch-4.1 Mar 31, 2026
31 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liaoxin01 @Thearas @hello-stephen @yiguolei