Uploading to S3 type storage chunked without knowing content length

[yakovm3]

I need to stream file to S3 type storage.

I know that V2 authorization do support chunked streaming using PUT without knowing the content length but Its not clear what the exact API and how exactly I implement such upload
I have found how to force V2 signer but not anything beyond that.
Note:I don't want to use multi-part upload.

Thanks

[zhangzhx]

Hi, the uploading (PutObject) currently doesn't support chunked encoding using SigV2. It is only feasible in SigV4 and you must enable body signing. However, SDK is not exactly respecting the Chunked Encoding specification that it still calculates the Content-Length header before sending the body to the wire.

What is the use case for your request? Do you just need to send chunked content without knowing the length or do you have to use SigV2? Can you use SigV4 for sending the chunked content?

[yakovm3]

Hеre the user says

I had been using native Java to implement the REST api with V2 signature for a while until I realize the "streaming" that seems to work blows up when my data size gets large. The reason is because Java will buffer ALL the data in order to compute its length. When my data exceeds the heap size, my app will die. The way to work around it is to use "Transfer-coding = chunked"

Did this functionality existed?
In the end I want to stream my data without knowing the size beforehand and without knowing part of each chunk i.e open a stream and write to the file on the server(not multi part)Does it possible?

[zhangzhx]

Hi @yakovm3,

I don't think this functionality exists at the moment. S3 has the specification that the Content-Length header must be provided.

I can go ahead ask the S3 service team whether they have a plan to fully support chunked encoding scheme where the Content-Length header could be eliminated.

By the way, the chunked encoding feature is only supported when you are using SigV4 and enabling body signing.

[debora-ito]

S3 upload for objects with unknown content-length is now available in v2 TransferManager, using the CRT-based S3 client.

Check the Java SDK v2 Developer Guide for more information and some code examples using v2 TransferManager. If you experience any issue or have any feedback, let us know in the v2 repo.

Reference:

• Announcing end-of-support for AWS SDK for Java v1.x effective December 31, 2025 - blog post

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[narengogi]

8 years later and this functionality still doesn't exist