lib-storage Upload working but result is missing

[bmichotte]

Describe the bug

Hello there,

Since 3.26.0 the Upload command from lib-storage does not return the same data and therefore is not usable

Edit: to clarify : the file is sent correctly to the bucket, but the result is not correct

Your environment

SDK version number

@aws-sdk/client-s3: 3.26.0 and 3.27.0
@aws-sdk/lib-storage: 3.26.0 and 3.27.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

▶ node -v
v16.2.0

Steps to reproduce

const client = new S3({
    region,
    credentials: {
        accessKeyId,
        secretAccessKey,
     },
 })
const params = {
    Bucket: bucket,
    Key: filename,
    Body: file,
    ACL: 'public-read',
    ContentType: type,
}

const upload = new Upload({
    client,
    queueSize: 3,
    params,
})
const data = await upload.done()

console.log({ data })

Observed behavior

With 3.26.0 and 3.27.0, the content of data is

{
  data: {
    '$metadata': {
      httpStatusCode: 200,
      requestId: undefined,
      extendedRequestId: 'dD4fCNw0SfXPpz6AKXc......',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    BucketKeyEnabled: undefined,
    ETag: '"8e79af4e71fa......."',
    Expiration: undefined,
    RequestCharged: undefined,
    SSECustomerAlgorithm: undefined,
    SSECustomerKeyMD5: undefined,
    SSEKMSEncryptionContext: undefined,
    SSEKMSKeyId: undefined,
    ServerSideEncryption: 'AES256',
    VersionId: 'OCh1_......'
  }
}

Expected behavior

with 3.25.0 the result is

{
  data: {
    '$metadata': {
      httpStatusCode: 200,
      requestId: undefined,
      extendedRequestId: 'H2cT3+QuQ3vG2/fBtj/GP8g...',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Bucket: 'The name of the bucket',
    BucketKeyEnabled: undefined,
    ETag: '"5a101d2..."',
    Expiration: undefined,
    Key: 'the key provided',
    Location: 'https://xxx.s3.eu-central-1.amazonaws.com/the key provided',
    RequestCharged: undefined,
    SSEKMSKeyId: undefined,
    ServerSideEncryption: 'AES256',
    VersionId: '28mF6wpfmGAe...'
  }
}

[ZimNovich]

Yes, I've encountered the same issue with 3.27.0:

2021-08-20T23:10:14.084466+00:00 app[web.1]: Result S3 is ...
2021-08-20T23:10:14.084859+00:00 app[web.1]: {
2021-08-20T23:10:14.084861+00:00 app[web.1]:   '$metadata': {
2021-08-20T23:10:14.084861+00:00 app[web.1]:     httpStatusCode: 200,
2021-08-20T23:10:14.084861+00:00 app[web.1]:     requestId: undefined,
2021-08-20T23:10:14.084862+00:00 app[web.1]:     extendedRequestId: 'W+LYk/slTx.....',
2021-08-20T23:10:14.084863+00:00 app[web.1]:     cfId: undefined,
2021-08-20T23:10:14.084863+00:00 app[web.1]:     attempts: 1,
2021-08-20T23:10:14.084864+00:00 app[web.1]:     totalRetryDelay: 0
2021-08-20T23:10:14.084864+00:00 app[web.1]:   },
2021-08-20T23:10:14.084864+00:00 app[web.1]:   BucketKeyEnabled: undefined,
2021-08-20T23:10:14.084865+00:00 app[web.1]:   ETag: '"4c4affea....."',
2021-08-20T23:10:14.084865+00:00 app[web.1]:   Expiration: undefined,
2021-08-20T23:10:14.084865+00:00 app[web.1]:   RequestCharged: undefined,
2021-08-20T23:10:14.084867+00:00 app[web.1]:   SSECustomerAlgorithm: undefined,
2021-08-20T23:10:14.084867+00:00 app[web.1]:   SSECustomerKeyMD5: undefined,
2021-08-20T23:10:14.084867+00:00 app[web.1]:   SSEKMSEncryptionContext: undefined,
2021-08-20T23:10:14.084868+00:00 app[web.1]:   SSEKMSKeyId: undefined,
2021-08-20T23:10:14.084868+00:00 app[web.1]:   ServerSideEncryption: undefined,
2021-08-20T23:10:14.084869+00:00 app[web.1]:   VersionId: undefined
2021-08-20T23:10:14.084869+00:00 app[web.1]: }

Below is what the version 3.25.0 returns:

2021-08-21T00:12:49.560869+00:00 app[web.1]: Result S3 is ...
2021-08-21T00:12:49.561834+00:00 app[web.1]: {
2021-08-21T00:12:49.561835+00:00 app[web.1]:   '$metadata': {
2021-08-21T00:12:49.561835+00:00 app[web.1]:     httpStatusCode: 200,
2021-08-21T00:12:49.561836+00:00 app[web.1]:     requestId: undefined,
2021-08-21T00:12:49.561837+00:00 app[web.1]:     extendedRequestId: 'fDuaa+78.....',
2021-08-21T00:12:49.561837+00:00 app[web.1]:     cfId: undefined,
2021-08-21T00:12:49.561838+00:00 app[web.1]:     attempts: 1,
2021-08-21T00:12:49.561838+00:00 app[web.1]:     totalRetryDelay: 0
2021-08-21T00:12:49.561839+00:00 app[web.1]:   },
2021-08-21T00:12:49.561839+00:00 app[web.1]:   Bucket: 'snowball12',
2021-08-21T00:12:49.561839+00:00 app[web.1]:   BucketKeyEnabled: undefined,
2021-08-21T00:12:49.561840+00:00 app[web.1]:   ETag: '"0a5d0......"',
2021-08-21T00:12:49.561840+00:00 app[web.1]:   Expiration: undefined,
2021-08-21T00:12:49.561841+00:00 app[web.1]:   Key: 'www.es.co.uk-20...',
2021-08-21T00:12:49.561841+00:00 app[web.1]:   Location: 'https://serve.s3.us-east-1.amazon.....',
2021-08-21T00:12:49.561841+00:00 app[web.1]:   RequestCharged: undefined,
2021-08-21T00:12:49.561842+00:00 app[web.1]:   SSEKMSKeyId: undefined,
2021-08-21T00:12:49.561842+00:00 app[web.1]:   ServerSideEncryption: undefined,
2021-08-21T00:12:49.561842+00:00 app[web.1]:   VersionId: undefined
2021-08-21T00:12:49.561843+00:00 app[web.1]: }

[samchungy]

It's related to #2605. This bit of code here:

aws-sdk-js-v3/lib/lib-storage/src/Upload.ts

Line 215 in 4463581

result = this.putResponse!;

Since this.putResponse PutObjectCommandOutput doesn't contain Key or Location.

That being said -> the return type for both Key and Location are optional on CompleteMultipartUploadCommandOutput.

Not sure how you would surface Location as the API for PutObject does not return it like CompleteMultipart does. However, Key in PutObjectCommandOutput could be surfaced similarly to how CompleteMultipart does it.

https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/protocols/Aws_restXml.ts#L5029

[Linkgoron]

@samchungy I assume that what's missing in my PR is the following code from the previous SDK:
https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js#L681

It contains exactly the three missing properties (from what I can tell) - Key, Bucket and Location

[erf]

Same. Location (the object URL) is missing when migrating from v2 -> v3. Tried both PutObjectCommand and Upload

[JanJakes]

Run into the same problem, see #2675.

[wallind]

Run into the same problem, see #2675.

ditto^. experiencing same problem trying to upgrade from v2 -> v3 for bundle size reduction benefits. Downgrading to 3.25.0 works for me as well as a temporary fix. Bumping to hopefully get this rectified.

[ajredniwja]

@samchungy thanks for the PR, I have had someone look at it, can you please address the comments. Thanks for your contribution as well.

[samchungy]

@samchungy thanks for the PR, I have had someone look at it, can you please address the comments. Thanks for your contribution as well.

@ajredniwja done.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.