SDK v3 upload has three times as many API calls vs v2 for some streams

[Linkgoron]

Describe the bug

We've recently upgraded to SDK v3 from SDK v2, and we've noticed that in some of our services, those using upload, the number of API calls has tripled vs SDK v2 for a lot of our smaller uploads, and obviously this also noticeably increased everything related to more API calls to S3. The reason is that SDK v2 upload has specific code for streams that finish before the upload starts, and uses a single PUT command to upload the file in one go. In SDK v3, lib-storage does not have this optimization, and always uses the create/upload/complete commands.

This kind of behaviour was very unexpected for us when we upgraded from an older version to a newer version, and we definitely did not expect a tripling of the number API calls. Is there a specific reason why this behavior was not ported to v3, or any reason that this will not get ported in the future?

Note that the upgrade guide claims that In v3, @aws-sdk/lib-storage package is available. It supports all the features offered in v2 upload(), and supports both Node.js and browsers runtime.

[ajredniwja]

Hey @Linkgoron thanks for bringing this up, can you please share your use case and the code you were using with v2 and v3? I have personally not encountered this.

[Linkgoron]

@ajredniwja Yes, we're using Multer and Multer-s3 to upload files using express, and we've forked Multer-s3 to use the lib-storage v3 sdk instead of the v2 sdk.

However, at least by a quick look at the sdk code (I did not look thoroughly, though), I think that just providing a stream that emits a single buffer would reproduce this.

[MoLow]

Hi @ajredniwja,
the use case is simple - upgrading from sdkV2 to sdkV3 resulted in tripling our s3 costs:

we can obviously make changes on our side to solve this, but as @Linkgoron said, a library upgrade isn't expected to lead such a change

[AllanZhengYP]

@Linkgoron @MoLow

Is the object you are uploading mostly smaller than 5mb? The API call graph indicates the difference: it previously uses putObject API but it switched to use multipart upload API later. In v2, when the object smaller than 5mb, the s3.upload() will use the putObject API by default. Only when the object is bigger than 5 mb will it use the multipart upload API.

Whereas in v3, you are using the multipart upload upload()directly, which will make StartMultipartUpload, UploadPart, and CompleteMultipartUpload API calls. It is intended for uploading bigger object, with ability to pause and resume the upload. In these case, S3's PutObject API is more preferrable over the upload().

I will create a feature request for falling back to use PutObject API when the object is smaller than certain threshold.

[Linkgoron]

@Linkgoron @MoLow

Is the object you are uploading mostly smaller than 5mb?

In general, for our use case, this code is generic on the server and doesn't know the size of the files that it uploads. However, it looks like the files are mostly small.

Whereas in v3, you are using the multipart upload upload()directly

Per the docs, s3.upload and the lib-storage upload should behave the same.

I will create a feature request for falling back to use PutObject API when the object is smaller than certain threshold.

Thanks. Note that I get that smaller files that are chunked should probably get uploaded using multi-part (if it takes time to send them to the server), but if the request has already transferred to the server (even though it's a stream), it should use PUT when possible just like aws-sdk v2 does.

[AllanZhengYP]

Closing this as the root cause has been tracked at #2593