Error Occurs When Closing File After Uploading Via S3.PutObject

[MikeMangialardi]

Title is pretty self-explanatory. I open a file and upload it to S3 using PutObject. After upload is complete, an "Invalid argument" error is thrown when I attempt to close the file. Sample code below will reproduce the issue:

package s3

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
    "os"
    "testing"
)

func TestStreaming(t *testing.T) {
    f, err := os.Open("/path/to/test/file.txt")
    checkErr(err, t, "error opening file")

    stat, err := f.Stat()
    checkErr(err, t, "error getting file info")

    conn := s3.New(session.New(), &aws.Config{Region: aws.String("us-east-1")})

    _, err = conn.PutObject(&s3.PutObjectInput{
        Bucket:        aws.String("bucketname"),
        Key:           aws.String("test/key"),
        Body:          f,
        ContentType:   aws.String("text/plain"),
        ContentLength: aws.Int64(stat.Size()),
    })
    checkErr(err, t, "error uploading file")

    err = f.Close()
    checkErr(err, t, "error closing file")
}

func checkErr(err error, t *testing.T, message string) {
    if err != nil {
        t.Log(message)
        t.Log(err)
        t.Fail()
    }
}

[xibz]

Hello @MikeMangialardi, thank you for reaching out to us. I will take a look at this and see what is going on. Thank you for providing some code to test with!

[xibz]

@MikeMangialardi, it looks like in net/http, the WriteBody method will always close the body. It looks like os.File cannot have multiple calls to Close. I don't believe this is an issue with the SDK, but a side effect of the net/http's transport.

[MikeMangialardi]

Interesting. You may want to change the interface that you accept for the Body parameter in that case. Because you accept an io.ReadSeeker, which does not have a Close() method,I assumed that your library wouldn't close the file. Either way, thank you for for your prompt and thorough response!

[jasdel]

Thanks for the feedback, @MikeMangialardi. PR #871, is looking to fix the underlying issue that you encountered. Specifically, that the SDK's retry logic allowed the raw io.ReadSeeker to make its way to the http.Request without being wrapped. Once this bug is fixed you'll be able to control when the src of the passed in io.ReadSeeker is closed, (aka your file). The change makes sure the http.Request won't close the reader unexpectedly.

[MikeMangialardi]

@jasdel Thanks for the response! That makes sense, and I will work around until that PR is approved and merged.

[xibz]

@MikeMangialardi - After reading my response, it made it seem like we weren't going to do anything about it. Yes, with the PR #871, this should resolve this issue. Sorry for the unclear response. It wasn't my intention!

[jasdel]

@MikeMangialardi I just merged #874, and this fixes the issue of the SDK's Request reader potentially encountering race condition. Let us know if you have any issues, or feedback.

You should be able to consistently close the file provided to SDK's requests now.