Skip to content

feat(services/s3): add multipart copier#7541

Merged
Xuanwo merged 1 commit into
mainfrom
xuanwo/copier-s3-multipart
May 18, 2026
Merged

feat(services/s3): add multipart copier#7541
Xuanwo merged 1 commit into
mainfrom
xuanwo/copier-s3-multipart

Conversation

@Xuanwo
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo commented May 17, 2026

Which issue does this PR close?

Part of #7521.

Rationale for this change

S3 CopyObject only supports copying source objects up to the single-request copy limit. Larger objects need multipart copy, but OpenDAL's Copier API should keep those backend details internal while still exposing progress and abort support.

What changes are included in this PR?

This adds a reusable oio::MultipartCopy / oio::MultipartCopier helper that mirrors the existing multipart writer pattern: services provide copy_once, initiate_copy, copy_part, complete_copy, and abort_copy, while the core helper owns upload state, concurrent part scheduling, completion, and abort.

S3 now uses that helper for copy operations. It stats the source to choose between single-request copy and multipart copy, declares copy_can_multi plus copy_multi_min_size / copy_multi_max_size, pins multipart copy to the observed source ETag/version when available, and aborts pending multipart uploads through the copier abort path.

This also adds CopyOptions::chunk as a best-effort execution hint for segmented copy. For S3, setting chunk chooses the multipart copy part size, clamped by the service copy-multi size capability, and makes objects at or above that chunk size use multipart copy. copy_with and copier_with both support chunk and concurrent.

Are there any user-facing changes?

S3 copy can now handle source objects that require multipart copy through the existing copy and copier APIs. Users can also set chunk / concurrent on copy_with to actively request segmented copy behavior without manually driving a Copier.

AI Usage Statement

AI assistance was used to help implement and validate this PR.

@Xuanwo Xuanwo marked this pull request as ready for review May 17, 2026 17:21
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels May 17, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 18, 2026
@Xuanwo Xuanwo force-pushed the xuanwo/copier-s3-multipart branch 3 times, most recently from c183f0a to b9c0aa8 Compare May 18, 2026 04:01
@Xuanwo Xuanwo force-pushed the xuanwo/copier-s3-multipart branch from b9c0aa8 to e59deb1 Compare May 18, 2026 04:07
@Xuanwo Xuanwo merged commit 8205993 into main May 18, 2026
454 checks passed
@Xuanwo Xuanwo deleted the xuanwo/copier-s3-multipart branch May 18, 2026 04:58
@Xuanwo Xuanwo mentioned this pull request May 18, 2026
13 tasks

match status {
StatusCode::OK => {
// S3 CopyObject may return an error embedded in a 200 OK response body
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed the bug but got lost in the PR...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants