-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3 PutObject Presigned URL #467
Comments
Hi @rayrutjes thanks for contacting us. Using presigned urls with the SDK is straightforward and only requires a few extra steps compared to using the API operations directly. A good strategy would be for your CLI client to send your service the file key, and MD5 checksum of each file to be uploaded to S3. Your service would then generate and return presigned URLs for each file. At a minimum the In your service the presigned URL will be generated for the bucket, key, and MD5 of the file to be uploaded. Your CLI should provide these fields to your service. The service would then return a presigned URL. svc := s3.New(session.New())
r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
r.HTTPRequest.Header.Set("Content-MD5", checksum)
url, err := r.Presign(15 * time.Minute)
if err != nil {
fmt.Println("error presigning request", err)
return
}
fmt.Println("URL", url) The CLI client would receive the URL and make a PUT request with the file content to upload. req, err := http.NewRequest("PUT", url, fileReader)
if err != nil {
fmt.Println("error creating request", url)
return
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Println("failed making request")
return
} |
@jasdel thank you so much, this is actually just what I needed. |
Great, glad to help. Let us know if you have any other questions, feedback, or ideas how we can improve the SDK. |
@jasdel I have implemented the suggested solution and also the md5 control but I keep getting the following error message back from s3: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<Error>
<Code>NotImplemented</Code>
<Message>A header you provided implies functionality that is not implemented</Message>
<Header>Transfer-Encoding</Header>
<RequestId>0A6E12B1B237E050</RequestId>
<HostId>gnpz4TUIqaWp67TsNZUpJu80Ll+XK+G55P+/qkLU/kiMH1yK/h2gUbsb9YIAtYqr12a9jSwGE98=</HostId>
</Error> Any clue? |
Thanks for getting back in touch with us @rayrutjes. Is the Transfer-Encoding header being added by your app by chance? The following is the example I tried running to reproduce the error you reported, But wasn't able to. I also tried setting Transfer-Encoding header explicitly and wasn't able to reproduce it that way either. Any additional script or example code you have would be very helpful in investigating this issue. package main
import (
"bytes"
"crypto/md5"
"encoding/base64"
"fmt"
"io"
"net/http"
"os"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
buf := bytes.NewReader(make([]byte, 10*1024*1024))
h := md5.New()
io.Copy(h, buf)
//
// Generate presigned PutObject URL
svc := s3.New(session.New())
r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String(os.Args[1]),
Key: aws.String(os.Args[2]),
})
r.HTTPRequest.Header.Set("Content-MD5", base64.StdEncoding.EncodeToString(h.Sum(nil)))
url, err := r.Presign(15 * time.Minute)
if err != nil {
fmt.Println("error presigning request", err)
return
}
fmt.Println("URL", url)
buf.Seek(0, 0)
//
// Use the presigned URL to put a object to S3
req, err := http.NewRequest("PUT", url, buf)
if err != nil {
fmt.Println("error creating request", url)
return
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Println("failed making request")
return
}
defer resp.Body.Close()
//
// Print out the response.
fmt.Println("Status", resp.StatusCode, resp.StatusCode)
o := &bytes.Buffer{}
io.Copy(o, resp.Body)
fmt.Println(o.String())
} |
@jasdel thank you again for your time, you put me in the right direction. I finally got it working. Cheers. |
@jasdel one last note for anyone trying to figure out the same kind of implementation. If you try to add ACL to the presigned url, you will also have to pass that exact same header to the request processing the actual upload. Is this actually a bug or a feature? |
The SDK intentionally will not hoist any One way we've considered improving this experience is to add an additional way to generate a presigning. But instead of just generating a URL a |
@jasdel I have to agree with you, a little helper function like the one available for adding the checksum to a request |
Hotdog, @rayrutjes, thanks for the tip on setting |
AWS s3 api does not accept Transfer-Encoding header, so we need to specify Content-Length explicitly For reference check out this: aws/aws-sdk-go#467 Fix: - releasing semaphore - last chank wrong size
AWS s3 api does not accept Transfer-Encoding header, so we need to specify Content-Length explicitly For reference check out this: aws/aws-sdk-go#467 Fix: - releasing semaphore - last chank wrong size - file.BatchStore
Hi there,
I would like a user to be able to upload a whole directory directly to S3 in a specific folder of a bucket.
The destination folder is dictated by my app, and the content sent by a cli on the client side.
I don't want the files to actually reach my server so I thing I want to pre-sign a PutObjectRequest for each file.
Is that actually possible with the sdk?
My cli is also written in go, how would I write the actual upload to the presigned url?
Otherwise, would a temporary IAM user (Temporary Security Credentials) per folder be more suited for this? It would avoid the roundtrip per file to obtain the presigned url.
Please point me to the correct resources if this is possible.
The text was updated successfully, but these errors were encountered: