feat(services/s3): add multipart copier#7541
Merged
Merged
Conversation
PsiACE
approved these changes
May 18, 2026
c183f0a to
b9c0aa8
Compare
b9c0aa8 to
e59deb1
Compare
dentiny
reviewed
May 22, 2026
|
|
||
| match status { | ||
| StatusCode::OK => { | ||
| // S3 CopyObject may return an error embedded in a 200 OK response body |
Member
There was a problem hiding this comment.
I fixed the bug but got lost in the PR...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #7521.
Rationale for this change
S3
CopyObjectonly supports copying source objects up to the single-request copy limit. Larger objects need multipart copy, but OpenDAL's Copier API should keep those backend details internal while still exposing progress and abort support.What changes are included in this PR?
This adds a reusable
oio::MultipartCopy/oio::MultipartCopierhelper that mirrors the existing multipart writer pattern: services providecopy_once,initiate_copy,copy_part,complete_copy, andabort_copy, while the core helper owns upload state, concurrent part scheduling, completion, and abort.S3 now uses that helper for copy operations. It stats the source to choose between single-request copy and multipart copy, declares
copy_can_multipluscopy_multi_min_size/copy_multi_max_size, pins multipart copy to the observed source ETag/version when available, and aborts pending multipart uploads through the copier abort path.This also adds
CopyOptions::chunkas a best-effort execution hint for segmented copy. For S3, settingchunkchooses the multipart copy part size, clamped by the service copy-multi size capability, and makes objects at or above that chunk size use multipart copy.copy_withandcopier_withboth supportchunkandconcurrent.Are there any user-facing changes?
S3 copy can now handle source objects that require multipart copy through the existing
copyandcopierAPIs. Users can also setchunk/concurrentoncopy_withto actively request segmented copy behavior without manually driving aCopier.AI Usage Statement
AI assistance was used to help implement and validate this PR.