[Feature] Cross-cluster replication for cloud-native table (Part-1: implementation in FE ) #60586
+532
−96
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: drake_wang <wxl250059@alibaba-inc.com>
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
alvin-celerdata
requested changes
Jul 4, 2025
| @@ -216,6 +217,7 @@ enum TTaskType { | |||
| REPLICATE_SNAPSHOT, | |||
| UPDATE_SCHEMA, | |||
| COMPACTION_CONTROL, | |||
| REPLICATE_LAKE_REMOTE_STORAGE, | |||
There was a problem hiding this comment.
All new added enum items should be added to the last to be compatible during upgrading.
There was a problem hiding this comment.
this is not needed, could reuse REPLICATE_SNAPSHOT
wxl24life
changed the title
| @@ -1922,6 +1923,11 @@ struct TTableReplicationRequest { | |||
| 7: optional i64 src_table_data_size | |||
| 8: optional map<i64, TPartitionReplicationInfo> partition_replication_infos | |||
| 9: optional string job_id | |||
| 10: optional Types.TRunMode src_cluster_run_mode | |||
There was a problem hiding this comment.
Could use src_table_type to tell whether the source is shared data or nothing, not need src_cluster_run_mode
Comment on lines
+426
to
+444
| struct TReplicateLakeRemoteStorageRequest { | ||
| 1: optional Types.TTransactionId transaction_id | ||
| 2: optional Types.TTableId table_id | ||
| 3: optional Types.TPartitionId partition_id | ||
| 4: optional Types.TTabletId tablet_id | ||
| 5: optional TTabletType tablet_type | ||
| 6: optional Types.TSchemaHash schema_hash | ||
| 7: optional Types.TVersion visible_version | ||
| 8: optional Types.TTabletId src_tablet_id | ||
| 9: optional TTabletType src_tablet_type | ||
| 10: optional Types.TSchemaHash src_schema_hash | ||
| 11: optional Types.TVersion src_visible_version | ||
| 12: optional binary encryption_meta | ||
| 13: optional Types.TVersion data_version | ||
| 14: optional Types.TTabletId faked_shard_id | ||
| 15: optional Types.TDatabaseId src_db_id | ||
| 16: optional Types.TTableId src_table_id | ||
| 17: optional Types.TPartitionId src_partition_id | ||
| } |
There was a problem hiding this comment.
I think you can add some fileds in the TReplicateSnapshotRequest and reuse it.
Using src_tablet_type to tell whether the source is shared data or nothing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
What I'm doing:
This PR introduces an online data migration solution tailored for shared-data clusters, complementing the earlier storage-coupled migration strategy.
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: