S3 Compatible Object Versioning - Beta

[iglesiasbrandon]

What is the problem/pain point?

Object Versioning is a key feature of S3-compatible object storage that Storj does not currently support. In order to attract and retain enterprise customers, object versioning needs to be supported.

What is Object Versioning?

Object versioning maintains multiple versions of every unique object in a bucket. Each time a new version of the same object is uploaded to the bucket, a new version is created. In addition, when an object is “deleted” a Delete Marker is added to the object rather than the execution of a hard delete. This allows for recovery of deleted objects in addition to recovery of previous versions of the object.

What is the impact?

Currently, Storj users have no way of keeping track of changes to objects in Storj DCS, and therefore customers have no ability to make use of the added benefits of versioning. Versioning would allow more control that users are already accustomed to having with other cloud storage providers and specifically enable use cases that require recovery from accidental deletion or overwriting of files.

Why now?

All major clouds (AWS, Google, Azure) support object versioning, so it is something all customers expect out of the box. As we begin to onboard enterprise customers with sophisticated use cases, closing this gap in S3 compatibility will bring us closer to our goal of being enterprise grade.

Links:

Milestones:

• Pre-Requisite Work: https://github.com/storj/storj/milestone/46
• Satellite: https://github.com/storj/storj/milestone/6
• S3 Compatibility: https://github.com/storj/gateway-mt/milestone/4

PRD:

• https://docs.google.com/document/u/1/d/1gZH5IWntGX-dcSsMSsgEeDGGQS-6emrppnksy9zZbkM/edit

Project information:
https://storjlabs.atlassian.net/wiki/spaces/ENG/pages/2608431105/Object+Versioning

Pre-Requisite Work

Beta Give feedback

1. New pending_objects table definition storj#6045
   +0
   
2. Adjust upload path to use pending_objects table storj#6046
   +0
   
3. Adjust listing pending objects for pending_objects/objects tables storj#6047
   +0
   
4. Check pending_objects and objects table while deleting bucket storj#6057
   +0
   
5. Adjust delete pending object to support pending_objects/objects tables storj#6048
   +0
   
6. Modify zombiedeletion chore to delete unused pending objects also from pending_objects storj#6050
   +0
   
7. Overwriting existing object is not removing old object segments storj#6255
   Bug +1
   
8. [Needs Verification] Adjust tally calculation to go through pending_objects too storj#6049
   +0
9. Check performance impact for CommitObject using both pending_objects/objects tables storj#6054
   +0
   
10. Enable feature flag to use pending_objects table storj#6053
    +0
11. Modify expireddeletion chore to delete expired objects also from pending_objects storj#6051
    +0
12. Drop code that was used to support pending_objects/objects tables at the same time storj#6052
    +0

Feature Development

Beta Give feedback

1. Object Versioning exploration (Time-box 2 Sprints) storj#5460
   +0
   
2. Object Versioning Blueprint storj#5808
   +0
   
3. Design for enabling/disabling object versioning storj#6127
   +0
   

S3 Compatibility

Beta Give feedback

1. Review MinIO's test suite (mint) to find potential improvements to requirements for Object Versioning gateway-mt#351
   Good First Issue edge s3-tests +3
   
2. Review Ceph's test suite to find potential improvements to requirements for Object Versioning edge#353
   Good First Issue edge s3-tests +3
   
3. PutObjectVersioning edge#355
   Needs Estimation Needs Grooming edge +3
4. ListObjectVersions edge#356
   Needs Estimation Needs Grooming edge +3

Bucket-Level Versioning Configuration

Beta Give feedback

1. Design for enabling/disabling object versioning storj#6127
   +0
   
2. Create bucket-level versioning DB schema storj#6385
   +0
   
3. Implement "feature flag" to allow enabling bucket-level versioning per project storj#6386
   +0
   
4. Implement support for enabling bucket-level versioning in uplink uplink#155
   +0
   
5. Add uplink command to allow user to enable/disable object versioning storj#6387
   Enhancement +1
   

[jtolio]

Comment from OKR tracking sheet:

Need to solidify what features we're going to implement. Strong acceptance criteria required.

[onionjake]

@ferristocrat The metainfo team is getting to the point that they can help out with this milestone. We need to start filling out the requirements, etc.

[iglesiasbrandon]

Initial PRD is on a google doc: https://docs.google.com/document/u/1/d/1gZH5IWntGX-dcSsMSsgEeDGGQS-6emrppnksy9zZbkM/edit

[storjrobot]

This issue has been mentioned on Storj Community Forum (official). There might be relevant details there:

https://forum.storj.io/t/a-trash-for-customers/22513/3

[ferristocrat]

ROM ROM: 10-12 sprints

[ferristocrat]

@amwolff - let's talk through Object Versioning next steps from that gateway side when you're back. I think at this point we should prioritize this over UploadPartCopy related tasks.

[iglesiasbrandon]

@onionjake fyi the project information document has information we should discuss in our leadership meetings.

[ferristocrat]

Oct 27 2023: Moving out due to other business priorities that were worked on.

[shaupt131]

Testing internally while we finish up a few priority fixes.

[iglesiasbrandon]

@amozoss do you think you could update our docs with information about object versioning?

[ferristocrat]

@amozoss do you think you could update our docs with information about object versioning?

I'm actually working on a doc. I'll create a PR this week in the docs repo

[iglesiasbrandon]

Closing this issue! Object versioning is in beta and any customer can enable it in the GUI.