AWS CLI uses streaming trailer chunked requests on newer versions

[halkyon]

I upgraded AWS CLI client and started getting MissingContentLength errors. Debugging the request, it seems like it's using STREAMING-UNSIGNED-PAYLOAD-TRAILER or STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER requests, but we don't support these.

PUT
/test-files/testfile.dat

content-encoding:aws-chunked
host:gateway.storjshare.io
transfer-encoding:chunked
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250121T032638Z
x-amz-decoded-content-length:1048576
x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-trailer:x-amz-checksum-crc64nvme

An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.

AWS CLI version: aws-cli/2.23.2 Python/3.12.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24.

I downgraded the client to the previous version (aws-cli/2.15.21 Python/3.11.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24 prompt/off) and it worked again. I guess AWS CLI decides to use the streaming payload trailer on newer versions. I couldn't find a workaround to turn it off, short of downgrading the client.

Test command: aws s3 --endpoint https://gateway.storjshare.io cp /tmp/testfile.dat s3://test-files/testfile.dat (1M file), and no config changes in ~/.aws/config. It seems any size file it tries to stream, and also the same for multi-part uploads.

References:

• AWS docs: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html
• recent minio fix: minio/minio@76913a9
• https://review.dev.storj.io/c/storj/minio/+/6842

[halkyon]

https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst

2.23.0
...
* feature:``s3``: The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the ``when_supported`` and ``when_required`` options - in the shared AWS config file using ``response_checksum_validation``, and as an env variable using ``AWS_RESPONSE_CHECKSUM_VALIDATION``.
...

Maybe this was the change?

[halkyon]

Artur and I discovered it changed in 2.23.0. It works in 2.22.35, which was the last version before it: https://github.com/aws/aws-cli/compare/2.22.35..2.23.0

[halkyon]

Related: aws/aws-cli#9214

[halkyon]

we also need to support the new checksum types, minio fixed this upstream here: minio/minio@827004c

[heunland]

see also boto/boto3#4392 <boto/boto3#4392
We now have at least 2 customers who have run into this issue, workaround is to downgrade boto3 to v1.35.99 for now.

[amwolff]

w.r.t. the MinIO fix: we need a clean room implementation of this and cannot be inspired by or even be looking at the MinIO's implementation. I've created some issues that represent this effort:

• support STREAMING-UNSIGNED-PAYLOAD-TRAILER and STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER #90
• support full object checksums #91
• support the CRC-64/NVME checksum algorithm type #92

We will assign someone who hasn't yet looked at these commits that Sean referenced.

[NiaStorj]

4/15/25:
Moving back to backlog, will pick up once the team is ready