[FEATURE] S3/Object Gateway Support - Experimental

[rebeccazzzz]

Is your feature request related to a problem? Please describe (👍 if you like this request)

Longhorn will support provisioning and managing s3 object gateways over longhorn volume to become an opinionated storage solution for having block/fs/object interface/protocol support.

Describe the solution you'd like

Support vendor-neutral S3 backend integration based on COSI protocol, https://github.com/kubernetes-sigs/container-object-storage-interface-api/tree/v0.1.0/crds (v1alpha1 has been released), so it's good for experimental integration.

The scope of COSI is only for bucket management, so for longhorn built in S3 backend, we need to have S3 instance management, bucket credentials management, etc.

Additional context

https://aquarist-labs.io/s3gw/
kubernetes/enhancements#1383

[mathieudruart]

does this feature could be related to this project ? : https://aquarist-labs.io/s3gw/

[innobead]

does this feature could be related to this project ? : https://aquarist-labs.io/s3gw/

Have u used that project? Any feedback?

[mathieudruart]

does this feature could be related to this project ? : https://aquarist-labs.io/s3gw/

Have u used that project? Any feedback?

Not yet, but we are planning to test it (not in production).

We already use Longhorn volumes for most of our PV/PVC and we are really interested for an S3 gateway on top of Longhorn. We found this feature request here and that existing project (that seems to use Ceph S3 gateway capabilities on top of Longhorn volumes), and we were just wondering if this project could not cover the need for this feature ?

[innobead]

Thanks for the info 👍 . Yes, it would be a choice for our object backend.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Object Storage (LEP 20230430) #6640
  The PR for the chart change is at: Object Storage (LEP 20230430) #6640

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR for the manager is at: Object Storage (LEP 20230430) longhorn-manager#2136
  The PR for the engine is at: Object Storage (LEP 20230430) longhorn-engine#934
  The PR for the instance-manager is at: Object Storage (LEP 20230430) longhorn-instance-manager#285
  The PR for the backupstore is at: Object Storage (LEP 20230430) backupstore#148

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at: LEP: Object Support for Longhorn #5832

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at: Object Storage (LEP 20230430) longhorn-ui#649

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at:
  The automation test case PR is at: https://github.com/longhorn/longhorn-tests/pull/1523/files
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at