Skip to content

[Enhancement](file-cache) Add fine-grained control for compaction file cache#61354

Open
gavinchou wants to merge 2 commits intoapache:branch-4.0from
gavinchou:gavin-pick-60609-4.0
Open

[Enhancement](file-cache) Add fine-grained control for compaction file cache#61354
gavinchou wants to merge 2 commits intoapache:branch-4.0from
gavinchou:gavin-pick-60609-4.0

Conversation

@gavinchou
Copy link
Contributor

Support selective caching of index files only during compaction in cloud mode.

Cherry-picked from #60609

Changes:

  • Add two mBool configs to control index-only caching for base and cumulative compaction:

    • enable_file_cache_write_base_compaction_index_only (default: false)
    • enable_file_cache_write_cumu_compaction_index_only (default: false)
  • Extend RowsetWriterContext with compaction_output_write_index_only field
    to mark whether only index files should be cached

  • Modify get_file_writer_options() to accept is_index_file parameter:

    • When compaction_output_write_index_only=true and is_index_file=false,
      set write_file_cache=false to skip caching data files
    • Index files continue to be cached normally
  • Update file writer creation call sites to pass is_index_file parameter:

    • Index file writers: pass true
    • Segment (data) file writers: pass false

Benefits:

  • Reduces cache pressure by avoiding caching large data files during compaction
  • Preserves index file caching for query performance
  • Provides separate control for base and cumulative compaction strategies
  • Maintains backward compatibility with default settings

…e cache

Support selective caching of index files only during compaction in cloud mode.

Changes:
- Add two mBool configs to control index-only caching for base and cumulative compaction:
  * enable_file_cache_write_base_compaction_index_only (default: false)
  * enable_file_cache_write_cumu_compaction_index_only (default: false)

- Extend RowsetWriterContext with compaction_output_write_index_only field
  to mark whether only index files should be cached

- Modify get_file_writer_options() to accept is_index_file parameter:
  * When compaction_output_write_index_only=true and is_index_file=false,
    set write_file_cache=false to skip caching data files
  * Index files continue to be cached normally

- Update file writer creation call sites to pass is_index_file parameter:
  * Index file writers: pass true
  * Segment (data) file writers: pass false

Benefits:
- Reduces cache pressure by avoiding caching large data files during compaction
- Preserves index file caching for query performance
- Provides separate control for base and cumulative compaction strategies
- Maintains backward compatibility with default settings

Cherry-picked from apache#60609

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@gavinchou gavinchou requested a review from yiguolei as a code owner March 16, 2026 02:45
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou
Copy link
Contributor Author

run buildall

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 87.50% (28/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.98% (19183/36205)
Line Coverage 36.17% (178753/494252)
Region Coverage 32.79% (138548/422522)
Branch Coverage 33.78% (60193/178191)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants