Skip to content

feat(copy): add recursive same-endpoint remote copy - #302

Merged
cxymds merged 1 commit into
mainfrom
cxymds/issue-1439-recursive-remote-copy
Jul 23, 2026
Merged

feat(copy): add recursive same-endpoint remote copy#302
cxymds merged 1 commit into
mainfrom
cxymds/issue-1439-recursive-remote-copy

Conversation

@cxymds

@cxymds cxymds commented Jul 23, 2026

Copy link
Copy Markdown
Member

Related issue

Closes rustfs/backlog#1439
Parent roadmap: rustfs/backlog#1370
Depends on merged multipart copy primitive PR #301.

Background and user impact

Recursive remote-to-remote copy previously stopped at the CLI boundary even though RustFS now exposes the required same-endpoint server-side copy primitives. Users could not safely copy complete prefixes without scripting individual object operations.

Solution

  • Traverse and paginate the complete source prefix before any mutation, then produce deterministic source-to-destination mappings.
  • Reject overlapping prefixes, self copies, duplicate destinations, missing pagination tokens, and repeated pagination tokens during preflight.
  • Reuse one S3 client pool per alias and the shared transfer executor for bounded concurrency, rate limiting, retry policy, and deterministic outcomes.
  • Route objects at or below 5 GiB through CopyObject and larger objects through multipart server-side copy with source identity checks.
  • Support dry run, continue-on-error, fail-empty, explicit skip-existing behavior, logical-byte progress, and deterministic summaries.
  • Preserve source/destination versions and multipart upload IDs in human results, and retain abort cleanup details after failures or cancellation.
  • Make cancellation race-safe across scheduling, rate waits, retry backoff, small CopyObject, and multipart cleanup; interrupted work is counted as cancelled and exits 130.
  • Add the RustFS beta.10 recursive-copy integration smoke test.

Compatibility

  • Cross-alias and cross-endpoint streaming copy remain out of scope.
  • Planned/bulk JSON output remains explicitly unsupported until a versioned output contract is introduced.
  • No protected CLI, JSON, exit-code, or configuration contract files are changed.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • CARGO_BUILD_JOBS=2 cargo test --workspace
  • Recursive remote-copy black-box suite: 11 tests, including 1001-object pagination, mapping, overlap, empty prefix, skip-existing, partial failure, 5 GiB multipart routing, abort cleanup, and SIGINT exit 130.
  • Flake check: recursive suite passed five consecutive full runs before final additions; final full workspace run passed with all 11 cases.

@cxymds
cxymds merged commit dea2407 into main Jul 23, 2026
17 checks passed
@cxymds
cxymds deleted the cxymds/issue-1439-recursive-remote-copy branch July 23, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant