serve s3: Allow Presigned Requests for Serving S3

[Wamy-Dev]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/presigned-keys-in-rclone-serve-s3/44449

What is your current rclone version (output from rclone version)?

rclone v1.65.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-33-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

What problem are you are trying to solve?

Allowing Presigned Keys to be used when serving S3. Using the S3 API you can generate a pre-signed URL just fine, but when it comes to actually using that pre-signed url, it fails with 403 Not Authorized Errors, disguised as a 400 Bad Request (Rclone logs it as 403, but responds with 400 on the url itself).

How do you think rclone should be changed to solve that?

Implementing authentication verification using the presigned URL. When a user goes to this type of link, it should check authentication from the AWSAcessKeyID and the Signature to see if it matches the Access Key and Secret Key (I don't know how it is calculated or verified, but I don't think S3 keeps track of it manually, I think it is some kind of hashing or something). I hope it isn't stored, as I have multiple S3 serves running to the same data located around the world for faster downloads, so it shouldn't just be stored on the server the presigned key request was sent to.

Link Example (not real)

https://storage.example.app/bucket/file.mkv?AWSAccessKeyId=vHNhZ9FZK124XXXXXXXXXXXXXwme0UDfSQAJdSDXXXXX0lduUqsh&Signature=Wb6YUQEMXXXXXXXXY8%3D&Expires=1707179008

Documentation about Presigned Keys

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html
• https://github.com/minio/minio/blob/7aa00bff898e6a44f452f6dba802af8d0d198374/cmd/signature-v2.go#L109
• https://github.com/minio/minio/blob/7aa00bff898e6a44f452f6dba802af8d0d198374/cmd/signature-v4.go#L207
• https://github.com/search?q=repo%3Aminio%2Fminio%20presign&type=code

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Bounty

I am also willing to put up a bounty to get this done as fast as possible as I use it for my business. Let me know how that process is done, or if it is even possible at all. Thanks!

[ncw]

I am also willing to put up a bounty to get this done as fast as possible as I use it for my business. Let me know how that process is done, or if it is even possible at all. Thanks!

If you want to drop an email to sales@rclone.com we can discuss! Thank you.

[jordanpadams]

Is there any status here as to when/if this new requirement will be implemented?

[Wamy-Dev]

nothing.

What I ended up doing is rolling my own storage API which verified created pre-signed keys using redis. Of course this is a ton of work, but ended up working better for me in the end.

[xuze1993]

nothing.

What I ended up doing is rolling my own storage API which verified created pre-signed keys using redis. Of course this is a ton of work, but ended up working better for me in the end.

just got the error in another gofakes3 api.

[ncw]

This could probably be implemented fairly simply in our gofakes3 fork if someone wants to have a go.

https://github.com/rclone/gofakes3

[Wamy-Dev]

not interested anymore, but using minio code would be simple, look at the links above if anyone wants to give it a shot. I'm rooting for you!

[itsHenry35]

This could probably be implemented fairly simply in our gofakes3 fork if someone wants to have a go.

https://github.com/rclone/gofakes3

alist-org/gofakes3@5799d51 You can refer to this commit in the fork I think, but I'm not going to give PR as I've already changed the link of the repo name in mine