Migrated v4 signing process to streaming. #37

Merged
merged 1 commit into from Mar 12, 2015

Conversation

Projects
None yet
3 participants
Member

kat-co commented Mar 12, 2015

  • Migrated v4 signing process to streaming. This limits the amount of data that needs to be stored in memory during the signing process.
@@ -309,16 +311,10 @@ func canonicalHeaders(sortedHeaderNames []string, host string, hdr http.Header)
// lowercase hexadecimal string.
func payloadHash(req *http.Request, hasher hasher) (string, error) {
@axw

axw Mar 12, 2015

Member

Probably should add a comment to this saying that it'll eat the request body.

@axw

axw Mar 12, 2015

Member

On second thoughts, I think a much simpler and less error prone approach would be to check in payloadHash if req.Body is an io.ReadSeeker. If it is, pass to hasher as-is and then seek back to the original pos, if it's not, use the old approach of reading into memory and then replacing req.Body.

@kat-co

kat-co Mar 12, 2015

Member

As discussed in IRC, http.NewRequest wraps all body parameters in an ioutil.NopCloser, so this type assertion won't work :(

s3/s3.go
@@ -201,14 +206,14 @@ func (b *Bucket) GetReader(path string) (rc io.ReadCloser, err error) {
// PutReader inserts an object into the S3 bucket by consuming data
// from r until EOF.
-func (b *Bucket) PutReader(path string, r io.Reader, length int64, contType string, perm ACL) error {
+func (b *Bucket) PutReader(path string, r io.ReadSeeker, length int64, contType string, perm ACL) error {
@axw

axw Mar 12, 2015

Member

Hrm. This is an API break, because someone could be relying on the function signature (e.g. taking a method value).

@dimitern

dimitern Mar 12, 2015

Member

+1 also the name doesn't match if it's a ReadSeeker. How about adding separate method?

@kat-co

kat-co Mar 12, 2015

Member

I cannot; the underlying object's stream must seek back to the beginning or there will be an error.

Member

dimitern commented Mar 12, 2015

Looks mostly OK, however I can't honestly approve it without some benchmarking before and after this change to see what's the impact.

Migrated v4 signing process to streaming.
- Migrated v4 signing process to streaming. This limits the amount of data that needs to be stored in memory during the signing process.

kat-co pushed a commit that referenced this pull request Mar 12, 2015

Merge pull request #37 from katco-/v4-signing-memory-fix
Migrated v4 signing process to streaming.

@kat-co kat-co merged commit 214aafd into go-amz:v3 Mar 12, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment