Skip to content

feat(copy-core): add multipart server-side copy primitive - #301

Merged
cxymds merged 1 commit into
mainfrom
cxymds/issue-1440-multipart-copy
Jul 23, 2026
Merged

feat(copy-core): add multipart server-side copy primitive#301
cxymds merged 1 commit into
mainfrom
cxymds/issue-1440-multipart-copy

Conversation

@cxymds

@cxymds cxymds commented Jul 23, 2026

Copy link
Copy Markdown
Member

Closes rustfs/backlog#1440

Parent roadmap: rustfs/backlog#1370
Depends on the version-aware object support delivered for rustfs/backlog#1369.

Background

The existing single-request copy path cannot copy objects above the S3 CopyObject limit. Recursive remote copy therefore needs a reusable multipart server-side copy primitive in the core ObjectStore boundary and the S3 backend.

Root cause

The core abstraction exposed only single-request copy. It had no multipart copy planner, typed cancellation path, per-part CopySourceRange support, or guaranteed abort handling for failures during an active multipart upload.

Solution

  • add a pure multipart copy planner with S3 size and part-count constraints
  • add multipart copy request/result, progress, and cooperative cancellation types to rc-core
  • expose multipart_copy through ObjectStore without leaking AWS SDK types into the CLI layer
  • implement CreateMultipartUpload, serial UploadPartCopy, CompleteMultipartUpload, and exactly-once AbortMultipartUpload cleanup in rc-s3
  • preserve source version IDs, source ETag preconditions, content type, user metadata, and destination encryption settings
  • map conditional conflicts and interruption to typed CLI semantics while redacting sensitive error context
  • add unit coverage for ranges, limits, errors, abort behavior, cancellation races, metadata, versioning, encryption, and progress
  • add a RustFS beta.10 integration smoke test that exercises more than one copy range

Cancellation boundary

Guaranteed cleanup uses the explicit cooperative MultipartCopyCancellation token. Directly dropping the entire future cannot run asynchronous abort cleanup; callers that need cancellation guarantees must signal the token and await the operation result.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • cargo test -p rustfs-cli --test integration --features integration --no-run

The follow-up rustfs/backlog#1439 will consume this primitive for recursive same-endpoint remote copy.

@cxymds
cxymds requested a review from overtrue July 23, 2026 05:29
@cxymds
cxymds merged commit e0ee4aa into main Jul 23, 2026
17 checks passed
@cxymds
cxymds deleted the cxymds/issue-1440-multipart-copy branch July 23, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant