Download Object from Pre-signed URL

[relvacode]

Is there functionality to download from a pre-signed URL?

I would like to use the nice concurrency features of the S3 downloader, except this requires full knowledge of the bucket auth details which my client doesn't know.

Perhaps I should use another library for concurrent generic file download? If so can you recommend one?

[jasdel]

Hi @relvacode the SDK doesn't currently posse the ability to download concurrently from a presigned URLs. Since the original presigned URL is generated for a specific operation and request it cannot be split into multiple byte ranges.

If you're the one generating the presigned URLs you could use the S3.GetObject operations with a range requests to generate your presigned URLs for the object you're wanting to download concurrently. You could distribute these multiple presigned URLs to your client and they would be able to split the request into multiple parts.

[jasdel]

HI @relvacode I'm going to close this issue for now. Please open it back up if you have any more question, or we can help clarify the issue you're facing.

[ghost]

Hi @jasdel - Reviving this old thread but can you help me understand how I can generate multiple presigned URLs from the GetObject API ?

[jasdel]

Hi @ananddhandhania the best way to get a presigned URL for the GetObject API call is to use the Request.Presign method of a Request.

svc := s3.New(sess)

req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
    Bucket: aws.String(myBucket),
    Key: aws.String(myKey),
})

urlStr, signedHeaders, err := req.PresignRequest(15 * time.Minute)

Once you have the URL string you can share the presigned URL and any HTTP client can use the URL to make a the request to the S3 service until the 15 minutes have expired.

In addition the signedHeaders returned need to also be set by the client that will make the request with the URL to ensure the request's signature is validated correctly.

[ghost]

Thanks @jasdel. Is this something unique to S3 sdk for go ?
I can't see something similar for java sdk.

[jasdel]

@ananddhandhania Each SDK should have a way to create presigned URLs for API operations. For the AWS SDK for Java checkout this document, http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html

[ghost]

Another approach I would like to share is this. As you can see, from a single presigned url, we can provide the bytes range. This means we can do a parallel download with raw curl by passing in the image bytes range in the http request header. Isn't this better than generating multiple presigned urls?

f45c89a59d13:~ ananddh$ curl -v -X GET -H "range: bytes=27-300" 'https://random-presigned-url-bucket.s3.amazonaws.com/very_large_image.jpg?AWSAccessKeyId=AADDSADASDASDAS&Expires=1493783501&x-amz-security-token=DADAsdasdsaasASDK3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDIRP9SbovN9s5vwG0SLTAaaag1edYamrIqoTZX340jaBixD6x3fyyZVtFKvTD1Gg3U3jornraaFXbyrqBxS0Vu5DeG%2BwWazISVEvyQt7gO6CogaN41KcylAO9y1qAPsGwsrw3P3UZa8u7n3XCmMx%2BOurWnLGeuCgQrdWbVhlOEQwYqBZGW%2BbPx90PR3LBjjuADThD2Qzfno%2B67ciTX3W1YLKCMmuhNxJonORSFE2GrjDlDqSasnD%2FsCS3pt0NTTMW04C8AxSXHP%2FwvQUQIaHk7djrs57ebx2xIUaP3cdN5VL8usoh72jyAU%3D&Signature=9k8a572lTpkipLu%2B8t%2FTLzoUSgE%3D'

• Trying 52.216.1.64...
• Connected to random-presigned-url-bucket.s3.amazonaws.com (52.216.1.64) port 443 (#0)
• TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• Server certificate: *.s3.amazonaws.com
• Server certificate: DigiCert Baltimore CA-2 G2
• Server certificate: Baltimore CyberTrust Root

GET /very_large_image.jpg?AWSAccessKeyId=ASIAJ7C2CSNM4SJX4PHA&Expires=1493783501&x-amz-security-token=FQoDYXdzEK3%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDIRP9SbovN9s5vwG0SLTAaaag1edYamrIqoTZX340jaBixD6x3fyyZVtFKvTD1Gg3U3jornraaFXbyrqBxS0Vu5DeG%2BwWazISVEvyQt7gO6CogaN41KcylAO9y1qAPsGwsrw3P3UZa8u7n3XCmMx%2BOurWnLGeuCgQrdWbVhlOEQwYqBZGW%2BbPx90PR3LBjjuADThD2Qzfno%2B67ciTX3W1YLKCMmuhNxJonORSFE2GrjDlDqSasnD%2FsCS3pt0NTTMW04C8AxSXHP%2FwvQUQIaHk7djrs57ebx2xIUaP3cdN5VL8usoh72jyAU%3D&Signature=9k8a572lTpkipLu%2B8t%2FTLzoUSgE%3D HTTP/1.1
Host: random-presigned-url-bucket.s3.amazonaws.com
User-Agent: curl/7.43.0
Accept: /
range: bytes=27-300

< HTTP/1.1 206 Partial Content
< x-amz-id-2: CMU8IDkjFAKF2qR/zqMQ9lP11bHiEMdhChKz2EuxsCcIjo6Nd7khGUrqo0Eq6eyV94+ggY+UH8k=
< x-amz-request-id: 56F1961D023C59BD
< Date: Tue, 02 May 2017 19:34:36 GMT
< Last-Modified: Wed, 09 Nov 2016 01:27:25 GMT
< ETag: "7a4c42317eaf4754e5217c1361c70dd5-9"
< x-amz-version-id: null
< Accept-Ranges: bytes
< Content-Range: bytes 27-300/71590911
< Content-Type: image/jpeg
< Content-Length: 274
< Server: AmazonS3
<

• Connection #0 to host random-presigned-url-bucket.s3.amazonaws.com left intact
  Or????(1?2-??'-??'Adobe Photoshop CS5 Macintosh2012:03:28 12:46:48????.?

[jasdel]

For download using a single presigned URL multiple times is a great way to go. The only consideration to make is that the presigned URL's expiration time is long enough that for large objects the presigned URL will not expire.

[tjumlani]

Can an object have multiple presigned urls ?

[jasdel]

Hi @tjumlani, yes, a presigned URL is just a reference a client can use to get/put an object. There is no limit to the number of presigned URLs that can reference a single object.

[tjumlani]

Thanks. I experimented on that a moment ago and has been validated by your answer.

[evdokim]

Sorry for opening an old thread, but I discovered that Java SDK have such functionality in their TransferManager: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html#download-com.amazonaws.services.s3.model.PresignedUrlDownloadRequest-java.io.File-

I am wondering is there a plan to support it in aws-sdk-go as well?