-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature]: allow to attach parts from a different disk (v2.0) #63087
Conversation
This is an automated comment for commit 20b2254 with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page
Successful checks
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's 90% copy-paste from localBackup.cpp
. Let's avoid copy-paste and make localBackup.cpp
work with two disks. We can pass src and dst disks to localBackup
as 2 separate arguments, so it will be able to check if it's the same disk or not. And if it's not the same, it will copy from one disk to another.
/// Used only when attach partition; Both for same disk and different disk. | ||
std::pair<MergeTreeData::MutableDataPartPtr, scope_guard> MergeTreeData::cloneAndLoadDataPart( | ||
const MergeTreeData::DataPartPtr & src_part, | ||
const String & tmp_part_prefix, | ||
const MergeTreePartInfo & dst_part_info, | ||
const StorageMetadataPtr & metadata_snapshot, | ||
const IDataPartStorage::ClonePartParams & params, | ||
const ReadSettings & read_settings, | ||
const WriteSettings & write_settings) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same, let's avoid copy-paste
Yes, it was broken in master for a while |
…aPart into a function, controlled by parameters; integrate remoteBackup and localBackup into a class, and change the name to Backup.
@tavplubix @alexey-milovidov Is there anything else that needs to be modified in this PR? Please let me know if you still have any questions. |
true, | ||
{}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add comments with the argument names (like /* max_level= */ {}
)
…intead_of_hardlinks param in Backup()
9f49c83
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Allow to attach parts from a different disk
Documentation entry for user-facing changes
Documentation is written (mandatory for new features)
Motivation:
Allow to attach parts from a different disk:
attach partition from the table on other disks using copy instead of hard link (such as instant table)
attach partition using copy when the hard link fails even on the same disk
mergetree and replicatedMergeTree Engineer can both support this feature
Example use:
The source table can be an instant table like here: https://github.com/ClickHouse/web-tables-demo
Create a destination table with the same structure with source table. Then we can run:
The destination table can attach the partition from source table even though the source table is an instant table.
Test
Additional information
This PR used to be merged in #60112. However, there are some problems after merged:
These problems have been fixed in this pr.#62283