Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 source should work for non-AWS S3 storages #5751

Closed
Phlair opened this issue Aug 31, 2021 · 2 comments · Fixed by #6398
Closed

S3 source should work for non-AWS S3 storages #5751

Phlair opened this issue Aug 31, 2021 · 2 comments · Fixed by #6398

Comments

@Phlair
Copy link
Contributor

Phlair commented Aug 31, 2021

Currently S3 source is tied to AWS S3 (with url strings etc.). Should be relatively straightforward to extend this to support other storage systems following S3 protocol.

@Phlair Phlair added type/enhancement New feature or request area/connectors Connector related issues labels Aug 31, 2021
@sherifnada
Copy link
Contributor

@Phlair what's an example of this? Minio? Any examples you add would help make this a reality :D

@Phlair
Copy link
Contributor Author

Phlair commented Sep 6, 2021

Some examples:

The ideal soultion would be provider agnostic, such that the code uses boto3 to handle s3 compatible storage systems in a single generic approach, rather than cases for each S3 provider and setting in spec. The generic approach would then just use a user-provided base url along with access and secret keys to hit the relevant storage (AWS could be default for base url perhaps)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants