feat(casbackends): support for s3 compatible endpoints (minio, cloudflare R2, ...)

[migmartri]

Adds support to registering an S3-compatible CAS-backend by providing a custom endpoint. making it compatible with Minio and Cloudflare R2

AWS S3 support has not changed, except that the region now is optional. Region is optional now because it doesn't make much sense in the concept of other providers.

chainloop cas-backend add aws-s3 --access-key-id AKIA5CHQ4CJGSHYWLEZF --secret-access-key ${SECRET_KEY} --bucket chainloop-test --name aws-test                     
┌──────────┬────────────────┬──────────┬─────────────┬───────────────┬─────────┐
│ NAME     │ LOCATION       │ PROVIDER │ DESCRIPTION │ LIMITS        │ DEFAULT │
├──────────┼────────────────┼──────────┼─────────────┼───────────────┼─────────┤
│ aws-test │ chainloop-test │ AWS-S3   │             │ MaxSize: 100M │ false   │
└──────────┴────────────────┴──────────┴─────────────┴───────────────┴─────────┘

Minio

chainloop cas-backend add aws-s3 --name minio-test --access-key-id asdivBQsihWhMU0L --secret-access-key REDACTED --bucket test-chainloop \
--endpoint http://localhost:19000 

┌────────────┬─────────────────────────────────────┬──────────┬─────────────┬───────────────┬─────────┐
│ NAME       │ LOCATION                            │ PROVIDER │ DESCRIPTION │ LIMITS        │ DEFAULT │
├────────────┼─────────────────────────────────────┼──────────┼─────────────┼───────────────┼─────────┤
│ minio-test │ http://localhost:19000/test-chainlo │ AWS-S3   │             │ MaxSize: 100M │ true    │
│            │ op                                  │          │             │               │         │
└────────────┴─────────────────────────────────────┴──────────┴─────────────┴───────────────┴─────────┘

Cloudflare R2

chainloop cas-backend add aws-s3 --access-key-id fa95fbfa470effd79fdbe28b61b0788e --secret-access-key REDACTED --bucket chainloop \
--endpoint https://35c24d45e1cd7bc36279b8a4d130c7fe.r2.cloudflarestorage.com --name cloud-flare-test
┌──────────────────┬─────────────────────────────────────┬──────────┬─────────────┬───────────────┬─────────┐
│ NAME             │ LOCATION                            │ PROVIDER │ DESCRIPTION │ LIMITS        │ DEFAULT │
├──────────────────┼─────────────────────────────────────┼──────────┼─────────────┼───────────────┼─────────┤
│ cloud-flare-test │ https://35c24d45e1cd7bc36279b8a4d13 │ AWS-S3   │             │ MaxSize: 100M │ true    │
│                  │ 0c7fe.r2.cloudflarestorage.com/chai │          │             │               │         │
│                  │ nloop                               │          │             │               │         │
└──────────────────┴─────────────────────────────────────┴──────────┴─────────────┴───────────────┴─────────┘

note that the location shows the custom endpoint provided during registration

Closes #1053

cc/ @hanygirgis

[migmartri]

the reason of storing the endpoint in the location instead of the credentials is so it's shown to the user and can't be updated later on.

[migmartri]

cloudflare doesn't support this feature.

[jiparis]

This is great. Thanks for tackling it.

[jiparis]

on another topic, does this do streaming already?

[migmartri]

The underlying library does, the service layer that servers bytestream does not.