Crt transfer manager fails when uploading file

[Lunkers]

Describe the bug

I'm trying to upload a file to an s3 bucket using the S3 SDK and crt library. Once the transfer finishes, I get the following error:
software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error. I'm able to upload the same file using the same credentials with the CLI, so I'm inclined to believe something might be wrong with the library.

Expected Behavior

The file should be transferred without issue.

Current Behavior

The transfer fails at 100% with the error software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates internal server error

Reproduction Steps

val uploadFileRequest = UploadFileRequest.builder()
                    .putObjectRequest {
                        it.bucket(bucketName).key("sub/file.mxf.abc")
                    }
                    .addTransferListener(LoggingTransferListener.create())
                    .source(Paths.get(fileLocation))
                    .build()

val fileUpload = transferManager.uploadFile(uploadFileRequest)

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.62

JDK version used

11

Operating System and version

Ubuntu 22.04.2 Jammy

[debora-ito]

@Lunkers does it happen every time?
Can you enable the verbose wirelogs? Make sure to redact any sensitive information.

[Lunkers]

No, it only happens consistently with a certain kind of xml-based file format. I'll provide verbose logs once back at work tomorrow.

[Lunkers]

Here are some logs for a failed request:

2023-05-10 09:51:05 | [TRACE] [2023-05-10T07:51:05Z] [00007ffaed2a8640] [S3MetaRequest] - id=0x7ff978095c10 Calling virtual meta request destroy function. |  
-- | -- | --
  |   | 2023-05-10 09:51:05 | [DEBUG] [2023-05-10T07:51:05Z] [00007ffaed2a8640] [S3MetaRequest] - id=0x7ff978095c10 Cleaning up meta request |  
  |   | 2023-05-10 09:51:05 | [DEBUG] [2023-05-10T07:51:05Z] [00007ffaed2a8640] [S3MetaRequest] - id=0x7ff978095c10 Meta request finished with error code 14341 (Response code indicates internal server error) |  
  |   | 2023-05-10 09:51:05 | [ERROR] [2023-05-10T07:51:05Z] [00007ffad6ffd640] [S3Client] - id=0x7ffb9059d420 Client could not retry request 0x7ffadc0029d0 for meta request 0x7ff978095c10 with token 0x7ffadc190290 due to error 1069 (Retry cannot be attempted because the maximum number of retries has been exceeded.) |  
  |   | 2023-05-10 09:51:05 | [DEBUG] [2023-05-10T07:51:05Z] [00007ffad6ffd640] [S3Client] - id=0x7ffb9059d420 Client scheduling retry of request 0x7ffadc0029d0 for meta request 0x7ff978095c10 with token 0x7ffadc190290. |  
  |   | 2023-05-10 09:51:05 | [ERROR] [2023-05-10T07:51:05Z] [00007ffad6ffd640] [S3MetaRequest] - id=0x7ff978095c10 Meta request failed from error 14341 (Response code indicates internal server error). (request=0x7ffadc0029d0, response status=500). Try to setup a retry. |  
  |   | 2023-05-10 09:51:05 | [DEBUG] [2023-05-10T07:51:05Z] [00007ffad6ffd640] [S3MetaRequest] - id=0x7ff978095c10: Request 0x7ffadc0029d0 finished with error code 14341 (aws-c-s3: AWS_ERROR_S3_INTERNAL_ERROR, Response code indicates internal server error) and response status 500 |  
  |   | 2023-05-10 09:51:05 | [TRACE] [2023-05-10T07:51:05Z] [00007ffad6ffd640] [S3MetaRequest] - id=0x7ff978095c10 Incoming body for request 0x7ffadc0029d0. Response status: 500. Data Size: 204. connection: 0x7ff907007470.

Not too much information here sadly, maybe the tracing doesn't log everything to the ID?

[Lunkers]

Regardless, it seems to be an issue with the server I'm trying to upload to rather than the SDK in this case, so I think we can close the issue.

EDIT: Maybe not actually. A colleague provided the following server logs:

2023-05-10T09:51:04.753104+02:00 <30.3> [SERVER]-1(id7) s3[5739]: NtStatusToStatus(1278) Unhandled error STATUS_INVALID_NETWORK_RESPONSE
2023-05-10T09:51:04.756086+02:00 <30.6> [SERVER]-1(id7) s3[5739]: [s3] PUT_OBJECT(1081951272): Failed with InternalError. An internal error was encountered. Please try again.
2023-05-10T09:51:04.756222+02:00 <30.6> [SERVER]-1(id7) s3[5739]: [s3] ERROR_RESPONSE(1081951276): Failed with InvalidRequest. Improper network request received.
2023-05-10T09:51:05.676773+02:00 <30.3> [SERVER]-1(id7) s3[5739]: NtStatusToStatus(1278) Unhandled error STATUS_INVALID_NETWORK_RESPONSE
2023-05-10T09:51:05.678835+02:00 <30.6> [SERVER]-1(id7) s3[5739]: [s3] PUT_OBJECT(1081951317): Failed with InternalError. An internal error was encountered. Please try again.
2023-05-10T09:51:05.678981+02:00 <30.6> [SERVER]-1(id7) s3[5739]: [s3] ERROR_RESPONSE(1081951322): Failed with InvalidRequest. Improper network request received.

@debora-ito does the sdk have some default encoding set when uploading? The files I'm trying to upload all contain UTF-8 text.

[Lunkers]

Some updates: I've tested the same method towards a different endpoint (on-prem minio), and received another error there (sha256 not matching); I started considering if the fact that the failing files have double file extensions was the problem, since file.mxf worked, but file.mxf.ppsc did not. However, I encountered the same error (internal server error) when trying to upload file.ppsc. Kind of unsure where to go from here.

[Lunkers]

We found that the issue was on the server side after all, some sort of bug with checksum validation. Disabling validation on our s3 client solved the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.