[S3 Client] Errors if the Body of PutObjectCommand is a Readable type.

[yaquawa]

Describe the bug

I got this error when I set the Body to a Readable type.

(node:88634) UnhandledPromiseRejectionWarning: NotImplemented: A header you provided implies functionality that is not implemented

SDK version number
3.13.1

To Reproduce (observed behavior)

import { fromIni } from '@aws-sdk/credential-provider-ini'
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import https from 'https'
import { Readable } from 'stream'
import awsConfig from './aws-exports'

const s3Client = new S3Client({
  credentials: fromIni({ profile: 'default' }),
  region: 'ap-northeast-1',
})

async function getFileFromUrl(url: string): Promise<Readable> {
  return new Promise((resolve) => {
    https.get(url, (response) => {
      resolve(response)
    })
  })
}

async function upload(file: Readable) {
  const uploadCommand = new PutObjectCommand({
    Bucket: awsConfig.aws_user_files_s3_bucket,
    Key: 'test.jpg',
    Body: file,
    ACL: 'public-read',
  })

  await s3Client.send(uploadCommand)
}

async function migrate() {
  const file = await getFileFromUrl(
    'https://example.com/logo.png'
  )
  await upload(file)
  console.log('done')
}

migrate()

I could confirm that it's ok if I change the Body to a string, so I'm sure I have the right permission to put an object to the bucket.

[guanzo]

Same here. From googling, the error is thrown when the ContentLength field is not passed. This seems like a regression because the v2 package aws-sdk works fine without the ContentLength field.

Working version: "aws-sdk": "^2.893.0"

    const { data: fileStream } = await axios.get(url, {
        responseType: 'stream',
    })

    const params = {
        Bucket: 'bucket',
        Key: 'key',
        Body: fileStream,
    }

    await s3.upload(params).promise()

Broken version: "@aws-sdk/client-s3": "^3.13.1"

    const { data: fileStream } = await axios.get(url, {
        responseType: 'stream',
    })

    const params = {
        Bucket: 'bucket',
        Key: 'key',
        Body: fileStream,
        // Uncommenting ContentLength makes it work.
        //ContentLength: 123,
    }

    await s3.send(new PutObjectCommand(params))

Error msg:

NotImplemented: A header you provided implies functionality that is not implemented                                                                                           
    at deserializeAws_restXmlPutObjectCommandError (C:\Users\Eric\workspace\arc-core\file-processor\node_modules\@aws-sdk\client-s3\dist\cjs\protocols\Aws_restXml.js:8158:41)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)                                                                                                       
    at async C:\Users\Eric\workspace\arc-core\file-processor\node_modules\@aws-sdk\middleware-serde\dist\cjs\deserializerMiddleware.js:6:20                                   
    at async C:\Users\Eric\workspace\arc-core\file-processor\node_modules\@aws-sdk\middleware-signing\dist\cjs\middleware.js:12:24                                            
    at async StandardRetryStrategy.retry (C:\Users\Eric\workspace\arc-core\file-processor\node_modules\@aws-sdk\middleware-retry\dist\cjs\defaultStrategy.js:56:46)           
    at async C:\Users\Eric\workspace\arc-core\file-processor\node_modules\@aws-sdk\middleware-logger\dist\cjs\loggerMiddleware.js:6:22                                        
    at async ingestFile (C:\Users\Eric\workspace\arc-core\file-processor\src\file-worker.js:110:24)                                                                           
    at async processFile (C:\Users\Eric\workspace\arc-core\file-processor\src\file-worker.js:59:13)                                                                           
    at async handleDocSnapshot (C:\Users\Eric\workspace\arc-core\file-processor\src\worker.js:126:9)                                                                          
    at async run (C:\Users\Eric\workspace\arc-core\file-processor\node_modules\p-queue\dist\index.js:255:29) {                                                                
  Code: 'NotImplemented',                                                                                                                                                     
  Header: 'Transfer-Encoding',                                                                                                                                                
  RequestId: 'QHA5TF1MGH479H3C',                                                                                                                                              
  HostId: 'bsSO2hMyHLaDlcSOL2kNlUvm8LsZtrQGn1pof0fedAkn3c2S0bTBH9v2MNU4BrEtgOCGosshRd4=',                                                                                     
  '$fault': 'client',                                                                                                                                                         
  '$metadata': {                                                                                                                                                              
    httpStatusCode: 501,                                                                                                                                                      
    requestId: undefined,                                                                                                                                                     
    extendedRequestId: 'bsSO2hMyHLaDlcSOL2kNlUvm8LsZtrQGn1pof0fedAkn3c2S0bTBH9v2MNU4BrEtgOCGosshRd4=',                                                                        
    cfId: undefined,                                                                                                                                                          
    attempts: 1,                                                                                                                                                              
    totalRetryDelay: 0                                                                                                                                                        
  }                                                                                                                                                                           
}                                                                                                                                                                             

[fabis94]

I'm running into the same issue. In v2 of the SDK you could easily upload streams of unknown length through s3.upload(), but doesn't look like this is possible with v3. The error message could be better as well, it could let you know that Content-Length is missing, instead of mentioning the Transfer-Encoding header which seems completely irrelevant.

Pretty big regression, I would say, just because of this I have to revert back to v2 of the SDK as there is no other way to easily upload streams of unknown length.

[AllanZhengYP]

Hi @fabis94 @yaquawa @guanzo, The error is indeed caused by stream length remaining unknown. We need to improve the error message and the documentation. As it's pointed out in the Upgrading Guide, the similar functionality as s3.upload() in v2 is now moved to @aws-sdk/lib-storage package.

[guanzo]

Aha, so this is intended behavior. I did not check the upgrade guide. Thanks for the update!

[ffxsam]

Just in case anyone needs a clear answer: putObject can't write a stream to S3, you have to use the Upload class in @aws-sdk/lib-storage.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.