Server Side Copy

[iglesiasbrandon]

Summary:

Users are unable to copy data within a project or bucket without downloading it and uploading it to the new location.

Pain Point:

Copying data without server-side copy is costly because the user must pay for the download bandwidth.

Intended Outcome

Users will have the ability to copy data from one location to another within a project without having to download and reupload it. This will enable users to move data within their projects and buckets without having to pay for egress bandwidth. This functionality will be added to the library's and CLI tool first and then incorporated into the satellite GUI later.

How will it work?

The technical details of how this functionality will work can be found on the blueprint.

Links:

Storj Milestone: https://github.com/storj/storj/milestone/5
Uplink Milestone: https://github.com/storj/uplink/milestone/1
Blueprint: https://review.dev.storj.io/c/storj/storj/+/5930
Test Plan: storj/storj#4317

[jtolio]

Note from OKR tracking spreadsheet:

Pending no design issues, there is a strong possibility this will be completed by end of Q1, but it will be tight.

[shaupt131]

Week 9 update:

• Design document is approved.
• Initial implementation of basic functionality is merged
  • Certain actions (such as deleting objects) still missing.
• Uplink implementation is ready, but not yet merged.
• Optimization work is in progress.
• Next steps
  • Enabling server-side copy on QA satellite

[shaupt131]

Week 11 update:

Satellite:

• Optimization work in progress
• Adjusting deletion of bucket

Uplink:

• Uplink implementation is merged
• Working on overriding metadata

Next Steps:

• Integrating with Uplink CLI binary
• Integrating with gateway

[shaupt131]

Week 13 update:

Satellite:

• Continued optimization work
• Finalizing deletion logic
• Finalizing ancestor references and management
• Solving unexpected CRDB changes that affected functionality.

Uplink:

• Determining and building test cases

Next Steps:

• Continue determining and building more test cases
• Working on overwriting destination on copy

[shaupt131]

Week 15 update:

• Deployed into production, but currently behind a feature flag.
  • Will be enabled on all satellites in the next release.
• Functionally 99% complete, including most relevant features
• Passed QA
• Gateway-ST version update ready for release

Next Steps:

• Product to write up a blog post
• Continue to determine and build more test cases, including testing in production
• Continue to assess what features will be added at a later date