Wrapper around s3 multipart upload protocol

[vsoch]

Has anyone given thought for how OCI could (easily? with some work?) be implemented to work alongside an s3 multipart upload compatible storage? For example, Minio is a good testing ground minio/minio#10813. The creation of the request is fairly straight forward, but then the request to upload a part (akin to the PATCH for a chunk) is quite different. https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/upload-part.html. I was thinking that it would be really useful to be able to implement the distribution spec alongside this one, but the challenge of course are the differences. The server could act as some receiver of the requests, parsing them and forwarding then to Minio (or the s3 compatible storage) but this is problematic in that it puts huge load on that server, which we don't want to do.

Does anyone have experience with allowing for this kind of support, or is the distribution spec really different enough that it's never going to be a thing?

[samuelkarp]

Amazon ECR's UploadLayerPart API is similar to S3's upload-part API.

The server could act as some receiver of the requests, parsing them and forwarding then to Minio (or the s3 compatible storage)

When you use Docker or a distribution-compatible client to push an image to ECR, the endpoint you push to does this. You can see the calls to UploadLayerPart in the CloudTrail logs in your AWS account.

I haven't considered trying to do something like this directly in the distribution spec, but I did implement an alternative containerd resolver that directly talks to ECR's API in place of going through the distribution-compatible endpoint.

[amouat]

It's probably worth pointing out that Docker Distribution supports S3 as a backend. It may be worth checking out how they do things: https://github.com/docker/distribution/blob/7d9f0677169fade147c0f99bda35c106e0d8a8fb/registry/storage/driver/s3-aws/s3.go