Skip to content

branch-4.0:[feature](cloud) Support compaction read-write separation for cloud mode #60310#61872

Closed
Yukang-Lian wants to merge 1 commit intoapache:branch-4.0from
Yukang-Lian:pick-60310-branch-4.0
Closed

branch-4.0:[feature](cloud) Support compaction read-write separation for cloud mode #60310#61872
Yukang-Lian wants to merge 1 commit intoapache:branch-4.0from
Yukang-Lian:pick-60310-branch-4.0

Conversation

@Yukang-Lian
Copy link
Copy Markdown
Collaborator

Summary

Cherry-pick from #60310 (commit 5cbe1b4) to branch-4.0

  • Implements compaction read-write separation for cloud mode so that only the cluster that last performed data load executes compaction, preventing file cache invalidation on read-only clusters
  • Default is disabled (enable_compaction_rw_separation = false), can be enabled per-cluster via BE config

Conflict Resolution

  • gensrc/proto/cloud.proto: TxnInfoPB field 21 (defer_deleting_pending_delete_bitmaps) does not exist on branch-4.0, only added load_cluster_id (field 22) from this PR

Key changes

  • Track "last active cluster" per tablet in TabletStatsPB (updated during commit_txn)
  • FE sets load_cluster_id in TxnInfoPB during begin_txn
  • BE background thread caches cluster status via get_cluster_status RPC
  • Skip compaction in get_topn_tablets_to_compact for tablets owned by other clusters
  • Safety valve: force compaction when version count exceeds 80% of max
  • Configurable takeover timeout when write cluster becomes unavailable

Test plan

  • Unit tests: cloud_cluster_info_test, compaction_rw_separation_test
  • Regression tests: test_compaction_rw_separation, test_compaction_cluster_takeover, test_compaction_rw_separation_disabled, test_compaction_rw_separation_force

@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Mar 30, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian Yukang-Lian changed the title [feature](cloud) Support compaction read-write separation for cloud mode #60310 branch-4.0:[feature](cloud) Support compaction read-write separation for cloud mode #60310 Mar 30, 2026
@Yukang-Lian
Copy link
Copy Markdown
Collaborator Author

run buildall

@doris-robot
Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 51.72% (30/58) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.16% (1789/2260)
Line Coverage 64.43% (31982/49641)
Region Coverage 65.26% (16011/24533)
Branch Coverage 55.84% (8523/15264)

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 22.22% (2/9) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 70.79% (143/202) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.97% (19227/36300)
Line Coverage 36.13% (179124/495760)
Region Coverage 32.77% (139035/424286)
Branch Coverage 33.71% (60310/178888)

@Yukang-Lian
Copy link
Copy Markdown
Collaborator Author

Superseded by #61920 (combined cherry-pick PR for branch-4.0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants