S3::Client - checksum errors not retried when doing Object#upload_file

[lukaszfiszer-monday]

Describe the bug

According to the documentation on Client#initialize :compute_checksums option "Checksum errors returned by Amazon S3 are automatically retried up to :retry_limit times".

However I have run into issues when uploading a file to S3 via Object#upload_file fail randomly with The Content-MD5 you specified did not match what we received. error. After debugging, I have found out that such failed requests are not retried but rather raise an exception immediately.

Expected Behavior

Checksum errors when uploading files to be retried according to :retry_limits and other related settings, just in docs.

Current Behavior

Checksum errors are not retried

Reproduction Steps

client = Aws::S3::Client.new(
  region: region_name,
  credentials: credentials,
  retry_limit: 3, # not necessary, since its the default 
  http_wire_trace: true # for debug output
)

# upload file with invalid content_md5. By default MD5 is calculated automatically by SDK.
Aws::S3::Object.new(bucket, key, client: client).upload_file(file_path, {content_md5: "QpS+Xllr5udn1RpvH6n80w=="})  

Observe output:

opening connection to XXXXXXXXX.s3.amazonaws.com:443...
opened
starting SSL for XXXXXX.s3.amazonaws.com:443...
SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256
<- "PUT /path/to/file.txt HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.112.0 ruby/2.7.2 x86_64-linux aws-sdk-s3/1.59.1\r\nContent-Md5: QpS+Xllr5udn1RpvH6n80w==\r\nExpect: 100-continue\r\nHost: XXXXXXXXXXXXXXX\r\nX-Amz-Date: 20221129T135635Z\r\nX-Amz-Content-Sha256: XXXXXXXXXXXXXXX\r\nAuthorization: XXXXXXXXXXXXXxXXXX, SignedHeaders=XXXXXXXXXXX, Signature=XXXXXXXXXXXXXXXX\r\nContent-Length: 26346\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 100 Continue\r\n"
-> "\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "x-amz-request-id: XXXXXXXXXXXXX\r\n"
-> "x-amz-id-2: XXXXXXXXX"
-> "Content-Type: application/xml\r\n"
-> "Transfer-Encoding: chunked\r\n"
-> "Date: Tue, 29 Nov 2022 13:56:35 GMT\r\n"
-> "Server: AmazonS3\r\n"
-> "Connection: close\r\n"
-> "\r\n"
-> "19e\r\n"
reading 414 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>4294be5e596be6e767d51a6f1fa9fcd3</ExpectedDigest><CalculatedDigest>QpS+Xllr5udn1RpvH6n70w==</CalculatedDigest><RequestId>VM9BCX8HGE18G0WB</RequestId><HostId>HeJ08teWmiKODC41BuvjK80zgQtfM44rflvVqIGUwmCpoZZiRSzIwQZaLzb3+0mV+yNfZIa8K4w=</HostId></Error>"
read 414 bytes
reading 2 bytes...
-> "\r\n"
read 2 bytes
-> "0\r\n"
-> "\r\n"
Conn close
Traceback (most recent call last):
        1: from (irb):280
Aws::S3::Errors::BadDigest (The Content-MD5 you specified did not match what we received.)

Possible Solution

Add a digest error here: https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/lib/aws-sdk-core/plugins/retries/error_inspector.rb#L40-L44

Additional Information/Context

A similar/identical issue #1264 was opened few years ago in v2, but closed without resolution

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3 - 1.59.0

Environment details (Version of Ruby, OS environment)

ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

[mullermp]

Thanks for opening an issue. You are correct that the documentation and actual behavior are not accurate. I'm asking around internally for some guidance here. We had decided not to retry response checksum failures, but this is for request and I'm not certain if this is something we should support or not. I think if we were to do this, we would need to add the digest error to error inspector and test that it works.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.