Cannot download a larger than 2GB from S3

[Mark1626]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
I'm not able to download a large >2GB file from s3. This seems to be because of a limitation with Buffer.alloc. Can you suggest a possible workaround for downloading

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?
Yes

Details of the browser/Node.js version
10.16.3

SDK version number
2.541.0

To Reproduce (observed behavior)

File in s3 is larger than 2GB

const s3ClientObject = new S3Client();
await s3ClientObject.getObject({
    Bucket: bucket,
    Key: key,
    Body: fileStream,
  }).promise();

Expected behavior
Should be able to download the file from s3

Screenshots

RangeError [ERR_INVALID_OPT_VALUE]: The value \"2536463704\" is invalid for option \"size\"
    at Function.alloc (buffer.js:278:3)
    at Object.alloc (/path/to/node_modules/aws-sdk/lib/util.js:136:28)
    at Object.concat (/path/to/node_modules/aws-sdk/lib/util.js:175:28)
    at Request.HTTP_DONE (/path/to/node_modules/aws-sdk/lib/event_listeners.js:396:36)
    at Request.callListeners (/path/to/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/path/to/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/path/to/node_modules/aws-sdk/lib/request.js:683:14)
    at IncomingMessage.onEnd (/path/to/node_modules/aws-sdk/lib/event_listeners.js:306:26)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)

Additional context

This seems to be because Buffer.alloc cannot create a buffer with size greater than (2^31)-1

https://nodejs.org/api/buffer.html#buffer_class_method_buffer_alloc_size_fill_encoding
https://nodejs.org/api/buffer.html#buffer_buffer_constants_max_length

cc @ajredniwja

[nullren]

@M4rk9696 you have to use multipart downloads to do this which getObject has support for

[ajredniwja]

Hey @M4rk9696,

Thank-you for reaching out, @nullren is right, you have to use multipart downloads for this.
Closing this issue, please reach out if you have any other questions.

[Mark1626]

@nullren @ajredniwja Thanks for suggestion, I'll check out the multipart download. Can this limit be specified in the documentation, it was sort of a gotcha when I was load-testing.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.