Failed to upload file using transfer manager to S3 bucket.

[as6432]

Describe the bug

After upgrading AWS CPP SDK to 1.11.178, file upload using transfer manager failing with error 'InvalidChunkSizeError Message: Only the last chunk is allowed to have a size less than 8192 bytes'.
We tried uploading files with various size but getting same error.

Expected Behavior

File upload using transfer manager should work.

Current Behavior

Failed to upload file of any size using aws transfer manager cpp APIs after upgrading aws cpp sdk 1.11.178 version.

Reproduction Steps

Take file of any size and try to upload it using aws cpp transfer manager APIs with AWS CPP SDK version 1.11.178.

Possible Solution

No response

Additional Information/Context

1116	15:38:25.830	 [8956]	(StorageArchive)	<9232>	EV:H	{AwsSdk::TransferManager} [Trace] Seeking input stream to determine content-length to upload file to bucket: janitacnonworm with key: 22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9
1117	15:38:25.830	 [8956]	(StorageArchive)	<9232>	EV:H	{AwsSdk::TransferManager} [Trace] Setting content-length to 60126 bytes. To upload file to bucket: janitacnonworm with key: 22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9
1118	15:38:25.830	 [8956]	(StorageArchive)	<9232>	EV:H	{AwsSdk::TransferManager} [Debug] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Scheduling a single-part upload.
1119	15:38:25.830	 [8956]	(StorageArchive)	<7520>	EV:H	{AwsSdk::TransferManager} [Info] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Updated handle status from [NOT_STARTED] to [IN_PROGRESS].
1120	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint str eval parameter: Region = ap-south-1
1121	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint bool eval parameter: UseFIPS = 0
1122	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint bool eval parameter: UseDualStack = 0
1123	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint bool eval parameter: UseArnRegion = 0
1124	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint bool eval parameter: DisableMultiRegionAccessPoints = 0
1125	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint str eval parameter: Bucket = janitacnonworm
1126	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Debug] Endpoint rules engine evaluated the endpoint: https://janitacnonworm.s3.ap-south-1.amazonaws.com
1127	15:38:25.830	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::Aws::Endpoint::DefaultEndpointProvider} [Trace] Endpoint rules evaluated props: {"authSchemes":[{"disableDoubleEncoding":true,"name":"sigv4","signingName":"s3","signingRegion":"ap-south-1"}]}
1145	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Note: Http payloads are not being signed. signPayloads=0 http scheme=https
1146	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Canonical Header String: amz-sdk-invocation-id:B56073AC-DD2A-44FA-8D33-42953E37C476|amz-sdk-request:attempt=1|content-encoding:aws-chunked|content-type:application/octet-stream|host:janitacnonworm.s3.ap-south-1.amazonaws.com|transfer-encoding:chunked|x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER|x-amz-date:20231017T100825Z|x-amz-decoded-content-length:60126|x-amz-meta-evmetadata:<EnterpriseVaultFile Version="1" FileIdentifier="610DC4D48ED73E1ABB1F5D05C8050CC1~28~66377FE9~00~1" FileType="DVSSP" ArchivedDateUTC="45216.4215277778" PartitionEntryId="122C863DFB567C94D9D818E5FE00F76851q10000EV-EVProduct-Server1"/>|x-amz-sdk-checksum-algorithm:CRC32|x-amz-storage-class:STANDARD|x-amz-trailer:x-amz-checksum-crc32|
1147	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Signed Headers value:amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-evmetadata;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer
1148	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Canonical Request String: PUT|/22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9|amz-sdk-invocation-id:B56073AC-DD2A-44FA-8D33-42953E37C476|amz-sdk-request:attempt=1|content-encoding:aws-chunked|content-type:application/octet-stream|host:janitacnonworm.s3.ap-south-1.amazonaws.com|transfer-encoding:chunked|x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER|x-amz-date:20231017T100825Z|x-amz-decoded-content-length:60126|x-amz-meta-evmetadata:<EnterpriseVaultFile Version="1" FileIdentifier="610DC4D48ED73E1ABB1F5D05C8050CC1~28~66377FE9~00~1" FileType="DVSSP" ArchivedDateUTC="45216.4215277778" PartitionEntryId="122C863DFB567C94D9D818E5FE00F76851q10000EV-EVProduct-Server1"/>|x-amz-sdk-checksum-algorithm:CRC32|x-amz-storage-class:STANDARD|x-amz-trailer:x-amz-checksum-crc32|amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-evmetadata;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer|STREAMING-UNSIGNED-PAYLOAD-TRAILER
1149	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Final String to sign: AWS4-HMAC-SHA256|20231017T100825Z|20231017/ap-south-1/s3/aws4_request|e7da88fddc3d3a32ff1b63e461b621cb45fd1dee8a37032f09be65c21904bccf
1150	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Final computed signing hash: e3bf44ceca50a065352725f9682438f916a79f43d14a2d3349de9c0b3c9c41e2
1151	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSAuthV4Signer} [Debug] Signing request with: AWS4-HMAC-SHA256 Credential=AKIA4TI2VRXXASSLYIYD/20231017/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-evmetadata;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=e3bf44ceca50a065352725f9682438f916a79f43d14a2d3349de9c0b3c9c41e2
1152	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSClient} [Debug] Request Successfully signed
1153	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Trace] Making PUT request to uri https://janitacnonworm.s3.ap-south-1.amazonaws.com/22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9
1154	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Info] Attempting to acquire connection for janitacnonworm.s3.ap-south-1.amazonaws.com:443
1155	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Debug] Pool doesn't exist for endpoint, creating...
1156	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Debug] Pool has no available existing connections for endpoint, attempting to grow pool.
1157	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinConnectionContainer} [Info] Pool grown by 2
1158	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Info] Connection now available, continuing.
1159	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Debug] Returning connection handle 05071580
1160	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Acquired connection 05071580
1161	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Warn] Failed setting TCP keep-alive interval with error code: 12018
1162	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpHttp2} [Debug] HTTP/2 enabled on WinHttp handle: 06404E40.
1163	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] AllocateWindowsHttpRequest returned handle 06404E40
1164	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] with headers:
1165	15:38:25.845	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] amz-sdk-invocation-id: B56073AC-DD2A-44FA-8D33-42953E37C476|amz-sdk-request: attempt=1|authorization: AWS4-HMAC-SHA256 Credential=AKIA4TI2VRXXASSLYIYD/20231017/ap-south-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-type;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-evmetadata;x-amz-sdk-checksum-algorithm;x-amz-storage-class;x-amz-trailer, Signature=e3bf44ceca50a065352725f9682438f916a79f43d14a2d3349de9c0b3c9c41e2|content-encoding: aws-chunked|content-type: application/octet-stream|host: janitacnonworm.s3.ap-south-1.amazonaws.com|transfer-encoding: chunked|user-agent: APN/1.0 Veritas/1.0 EnterpriseVault/15.0 api/S3 ft/s3-transfer|x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER|x-amz-date: 20231017T100825Z|x-amz-decoded-content-length: 60126|x-amz-meta-evmetadata: <EnterpriseVaultFile Version="1" FileIdentifier="610DC4D48ED73E1ABB1F5D05C8050CC1~28~66377FE9~00~1" FileType="DVSSP" ArchivedDateUTC="45216.4215277778" PartitionEntryId="122C863DFB567C94D9D818E5FE00F76851q10000EV-EVProduct-Server1"/>|x-amz-sdk-checksum-algorithm: CRC32|x-amz-storage-class: STANDARD|x-amz-trailer: x-amz-checksum-crc32|
1166	15:38:25.892	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 8200 bytes transferred for part [1].
1167	15:38:25.892	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 16400 bytes transferred for part [1].
1168	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 24600 bytes transferred for part [1].
1169	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 32800 bytes transferred for part [1].
1170	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 41000 bytes transferred for part [1].
1171	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 49200 bytes transferred for part [1].
1172	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 57400 bytes transferred for part [1].
1173	15:38:25.908	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] 60252 bytes transferred for part [1].
1174	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Received response code 403
1175	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Received content type application/xml
1176	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Received headers:
1177	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] HTTP/1.1 403 Forbidden|Connection: close|Date: Tue, 17 Oct 2023 10:08:26 GMT|Transfer-Encoding: chunked|Content-Type: application/xml|Server: AmazonS3|x-amz-request-id: CQY4E20AM11S74NR|x-amz-id-2: p/ji6GAfS1LdXZlvAvMIVSo1Icng7f28q93dO+1x7oK3XkaqflJGWMos+OABQAny+ZDu3qpXfek=|
1178	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Closing http request handle 06404E40
1179	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpSyncHttpClient} [Debug] Releasing connection handle 05071580
1180	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::WinHttpConnectionPoolMgr} [Debug] Releasing connection to endpoint janitacnonworm.s3.ap-south-1.amazonaws.com:443
1181	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSClient} [Debug] Request returned error. Attempting to generate appropriate error codes from response
1182	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSErrorMarshaller} [Trace] Error response is <?xml version="1.0"?>|<?xml version="1.0" encoding="UTF-8"?>|<Error>|    <Code>InvalidChunkSizeError</Code>|    <Message>Only the last chunk is allowed to have a size less than 8192 bytes</Message>|    <Chunk>2</Chunk>|    <BadChunkSize>8184</BadChunkSize>|    <RequestId>CQY4E20AM11S74NR</RequestId>|    <HostId>p/ji6GAfS1LdXZlvAvMIVSo1Icng7f28q93dO+1x7oK3XkaqflJGWMos+OABQAny+ZDu3qpXfek=</HostId>|</Error>|
1183	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSErrorMarshaller} [Warn] Encountered Unknown AWSError 'InvalidChunkSizeError': Only the last chunk is allowed to have a size less than 8192 bytes
1184	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSXmlClient} [Error] HTTP response code: 403|Resolved remote host IP address: |Request ID: CQY4E20AM11S74NR|Exception name: InvalidChunkSizeError|Error message: Unable to parse ExceptionName: InvalidChunkSizeError Message: Only the last chunk is allowed to have a size less than 8192 bytes|7 response headers:|connection : close|content-type : application/xml|date : Tue, 17 Oct 2023 10:08:26 GMT|server : AmazonS3|transfer-encoding : chunked|x-amz-id-2 : p/ji6GAfS1LdXZlvAvMIVSo1Icng7f28q93dO+1x7oK3XkaqflJGWMos+OABQAny+ZDu3qpXfek=|x-amz-request-id : CQY4E20AM11S74NR
1185	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSClient} [Warn] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
1186	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::AWSClient} [Debug] Server time is Tue, 17 Oct 2023 10:08:26 GMT, while client time is Tue, 17 Oct 2023 10:08:25 GMT
1187	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Error] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Failed to upload object to Bucket: [janitacnonworm] with Key: [22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9] HTTP response code: 403|Resolved remote host IP address: |Request ID: CQY4E20AM11S74NR|Exception name: InvalidChunkSizeError|Error message: Unable to parse ExceptionName: InvalidChunkSizeError Message: Only the last chunk is allowed to have a size less than 8192 bytes|7 response headers:|connection : close|content-type : application/xml|date : Tue, 17 Oct 2023 10:08:26 GMT|server : AmazonS3|transfer-encoding : chunked|x-amz-id-2 : p/ji6GAfS1LdXZlvAvMIVSo1Icng7f28q93dO+1x7oK3XkaqflJGWMos+OABQAny+ZDu3qpXfek=|x-amz-request-id : CQY4E20AM11S74NR
1188	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Debug] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Setting part [1] to [FAILED].
1189	15:38:25.923	 [8956]	(StorageArchive)	<5316>	EV:H	{AwsSdk::TransferManager} [Info] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Updated handle status from [IN_PROGRESS] to [FAILED].
1190	15:38:25.923	 [8956]	(StorageArchive)	<9232>	EV:H	{AwsSdk::TransferManager} [Info] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Attempting to abort multipart upload.
1191	15:38:25.923	 [8956]	(StorageArchive)	<9232>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle ID [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Cancelling transfer.
1192	15:38:25.923	 [8956]	(StorageArchive)	<9232>	EV:H	{CS3Storage::WriteObjectToStorage:#693} Error while uploading object. Exception [HTTP response code: 403|Resolved remote host IP address: |Request ID: CQY4E20AM11S74NR|Exception name: InvalidChunkSizeError|Error message: Unable to parse ExceptionName: InvalidChunkSizeError Message: Only the last chunk is allowed to have a size less than 8192 bytes|7 response headers:|connection : close|content-type : application/xml|date : Tue, 17 Oct 2023 10:08:26 GMT|server : AmazonS3|transfer-encoding : chunked|x-amz-id-2 : p/ji6GAfS1LdXZlvAvMIVSo1Icng7f28q93dO+1x7oK3XkaqflJGWMos+OABQAny+ZDu3qpXfek=|x-amz-request-id : CQY4E20AM11S74NR]
1193	15:38:25.923	 [8956]	(StorageArchive)	<7064>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Waiting on handle to abort upload. In Bucket: [janitacnonworm] with Key: [22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9] with Upload ID: [].
1194	15:38:25.923	 [8956]	(StorageArchive)	<7064>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Finished waiting on handle. In Bucket: [janitacnonworm] with Key: [22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9] with Upload ID: [].
1195	15:38:25.939	 [8956]	(StorageArchive)	<7064>	EV:H	{AwsSdk::TransferManager} [Trace] Transfer handle [07F66FE0-4F4F-40F2-B2AB-CE67DECB8B49] Status changed to FAILED after waiting for cancel status. In Bucket: [janitacnonworm] with Key: [22C863DFB567C94D9D818E5FE00F7685/2023/10-17/a/54f/a54f205d-becf-4d34-91ef-d4688f2812e9] with Upload ID: [].

AWS CPP SDK version used

1.11.178

Compiler and Version used

Visual Studio 2017

Operating System and version

Windows Server 2016

[sbiscigl]

Take file of any size and try to upload it using aws cpp transfer manager APIs with AWS CPP SDK version 1.11.178.

im able to run transfer manager to upload files with the following code

#include <aws/core/Aws.h>
#include <aws/core/utils/threading/Executor.h>
#include <aws/transfer/TransferManager.h>
#include <aws/s3/S3Client.h>
#include <iostream>
#include <thread>

using namespace std;
using namespace Aws;
using namespace Aws::Utils;
using namespace Aws::S3;
using namespace Aws::Transfer;

int main() {
  Aws::SDKOptions options;
  options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;

  Aws::InitAPI(options);
  {
    S3ClientConfiguration config({}, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never);
    auto s3Client = Aws::MakeShared<Aws::S3::S3Client>("S3Client", config);
    auto executor = Aws::MakeShared<Aws::Utils::Threading::PooledThreadExecutor>("executor", std::thread::hardware_concurrency() - 1);
    Aws::Transfer::TransferManagerConfiguration managerConfiguration(executor.get());
    managerConfiguration.s3Client = s3Client;
    managerConfiguration.checksumAlgorithm = S3::Model::ChecksumAlgorithm::NOT_SET;
    managerConfiguration.transferInitiatedCallback = [](const TransferManager *,
        const std::shared_ptr<const TransferHandle> &) -> void {};
    managerConfiguration.errorCallback = [](const TransferManager *,
        const std::shared_ptr<const TransferHandle> &,
        const Aws::Client::AWSError<Aws::S3::S3Errors> &error) -> void {
        std::cout << "Transfer failed!\n";
        std::cout << "Error: " << error.GetMessage() << "\n";
    };
    auto transferManager = Aws::Transfer::TransferManager::Create(managerConfiguration);

    transferManager->UploadFile("/your/file.txt",
        "yourebucket",
        "keyname.txt",
        "text/plain",
        {});

    transferManager->WaitUntilAllFinished();
  }
  Aws::ShutdownAPI(options);
  return 0;
}

in addition I ran all of the transfer manager tests and they seem to be passing. Can you give me any more details on what you are trying to upload or about the setup of your environment so i could try to replicate it?

[as6432]

We are trying to upload file using stream(Not passing file directly to API) from Windows machine. Also we are providing metadata to transfer manager.

Also we are creating S3client object additionally providing S3EndPointProvider object.

Please confirm and let me know in this case does it works at your end ?

[sbiscigl]

Please confirm and let me know in this case does it works at your end ?

could you provide a code example? perhaps editing the one above to create a scenario that fails for you? if you are able to provide code that fails it goes a long way for me to replicate it otherwise i'm kind of in the dark looking for the issue.

[as6432]

@sbiscigl I have uploaded code with AWS CPP SDK version 1.9. 257 and 1.11.178 in this (https://github.com/as6432/AWS_S3/blob/main/TransferManagerIssue.zip) zip file. you can open and update project to change SDK version by modifying AwsCPPSDKVersion parameter value in AWSS3.vcxproj. Also provide aws access key, secret key, bucket name, file path and region in main.cpp file.

On AWS SDK 1.9. 257 version it works fine but same code files for AWS SDK 1.11.178 version. Please let me know if you need additional information from me.

[as6432]

@sbiscigl Any update on this issue ?

[sbiscigl]

@as6432 Im hesitant about downloading a zip file for security reasons. is it possible to post just the plain cpp file on this issue? preferably just editing my earlier code.

[as6432]

@sbiscigl pasted code from above zip. Please try this code.

main.cpp

#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/transfer/TransferManager.h>
#include <aws/s3/model/GetBucketLocationRequest.h>
#include <aws/s3/model/GetBucketLocationResult.h>
#include <iostream>
#include "UnbufferedStreamSyncBuf.h"

using namespace Aws::S3;
using namespace std;

static const CStringA EV_METADATA_KEY = "evmetadata";
// Change below parameter as per your environment
static const char* AWS_ACCESS_KEY_ID        = "ACCESS_KEY";
static const char* AWS_SECRET_ACCESS_KEY    = "SECRET_ACCESS_KEY";
static const char* BUCKET                   = "BUCKET";
static const char* AWS_BUCKET_REGION        = "REGION";
static const char* FILE_PATH_TO_UPLOAD      = "E:\\POCs\\TransferManagerIssue\\InputFiles\\IBMNotesInstall.log";


#pragma region Internal Error Codes
#define CLOUD_E_INVALID_CREDENTIALS         0xC00471CFL
#define CLOUD_E_AWS_BUCKET_NOT_FOUND        0xC00471D0L
#define CLOUD_E_NETWORK_CONNECTIVITY_ISSUE  0xC00471D1L
#define STORAGE_E_ACCESS_DENIED             0xC0041801L
#define CLOUD_E_AWS_RESOURCE_NOT_FOUND      0xC00471D2L
#pragma endregion  

#pragma region Internal Methods

HRESULT GenerateStoreIdentifier(CStringA &csUUID)
{
#pragma comment(lib,"Rpcrt4.lib")

    GUID Uuid;
    if (RPC_S_UUID_NO_ADDRESS == ::UuidCreate(&Uuid))
    {
        return HRESULT_FROM_WIN32(RPC_S_UUID_NO_ADDRESS);
    }

    // Convert the UUID to a string
    unsigned char *StringUuid(NULL);
    if (RPC_S_OUT_OF_MEMORY == ::UuidToStringA(&Uuid, &StringUuid))
    {
        return HRESULT_FROM_WIN32(RPC_S_OUT_OF_MEMORY);
    }

    if (!StringUuid)
    {
        return E_POINTER;
    }

    csUUID = StringUuid;
    ::RpcStringFreeA(&StringUuid);

    return S_OK;
}

HRESULT MapAWSErrorsToHResult(const std::shared_ptr<Aws::Transfer::TransferHandle> &outcome, CStringW& csErrorDescription)
{
    Aws::StringStream ss;
    ss << outcome->GetLastError();
    csErrorDescription = ss.str().c_str();
    Aws::S3::S3Errors errType = outcome->GetLastError().GetErrorType();

    HRESULT hr = S_OK;
    switch (errType)
    {
    case Aws::S3::S3Errors::INVALID_ACCESS_KEY_ID:
    case Aws::S3::S3Errors::INVALID_SIGNATURE:
    case Aws::S3::S3Errors::SIGNATURE_DOES_NOT_MATCH:
        hr = CLOUD_E_INVALID_CREDENTIALS;
        break;
    case Aws::S3::S3Errors::NO_SUCH_BUCKET:
        hr = CLOUD_E_AWS_BUCKET_NOT_FOUND;
        break;
    case Aws::S3::S3Errors::NETWORK_CONNECTION:
        hr = CLOUD_E_NETWORK_CONNECTIVITY_ISSUE;
        break;
    case Aws::S3::S3Errors::ACCESS_DENIED:
        hr = STORAGE_E_ACCESS_DENIED;
        break;
    case Aws::S3::S3Errors::RESOURCE_NOT_FOUND:
    case Aws::S3::S3Errors::NO_SUCH_KEY:
        hr = CLOUD_E_AWS_RESOURCE_NOT_FOUND;
        break;
    default:
        hr = E_FAIL;
        break;
    }
    return hr;
}

template <typename R, typename E>
HRESULT MapAWSErrorsToHResult(const Aws::Utils::Outcome <R, E>& outcome, CStringW& csErrorDescription)
{
    csErrorDescription = outcome.GetError().GetMessage().c_str();
    Aws::S3::S3Errors errType = outcome.GetError().GetErrorType();

    HRESULT hr = S_OK;
    switch (errType)
    {
    case Aws::S3::S3Errors::INVALID_ACCESS_KEY_ID:
    case Aws::S3::S3Errors::INVALID_SIGNATURE:
    case Aws::S3::S3Errors::SIGNATURE_DOES_NOT_MATCH:
        hr = CLOUD_E_INVALID_CREDENTIALS;
        break;
    case Aws::S3::S3Errors::NO_SUCH_BUCKET:
        hr = CLOUD_E_AWS_BUCKET_NOT_FOUND;
    break;
    case Aws::S3::S3Errors::NETWORK_CONNECTION:
        hr = CLOUD_E_NETWORK_CONNECTIVITY_ISSUE;
        break;
    case Aws::S3::S3Errors::ACCESS_DENIED:
        hr = STORAGE_E_ACCESS_DENIED;
        break;
    case Aws::S3::S3Errors::RESOURCE_NOT_FOUND:
    case Aws::S3::S3Errors::NO_SUCH_KEY:
        hr = CLOUD_E_AWS_RESOURCE_NOT_FOUND;
        break;
    default:
        hr = E_FAIL;
        break;
    }
    return hr;
}

#pragma endregion

HRESULT WriteObjectToStorage(const CComPtr<IStream>& spStream,
    const CStringA& csMetadata,
    ULONGLONG nTotalBytesInput,
    ULONGLONG& nTotalBytesWritten,
    CStringW& csErrorDescription)
{
    HRESULT hr = S_OK;
    CStringA csKey;
    GenerateStoreIdentifier(csKey);

    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        Aws::String userName = AWS_ACCESS_KEY_ID;
        Aws::String password = AWS_SECRET_ACCESS_KEY;
        Aws::Auth::AWSCredentials creds(userName, password);

        // Get bucket's region
        Aws::Client::ClientConfiguration configuration;
        configuration.region = "us-east-1";
        S3Client client(creds, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
        Aws::S3::Model::GetBucketLocationRequest getBucketLocationRequest;
        getBucketLocationRequest.SetBucket(Aws::String(BUCKET));
        Aws::S3::Model::GetBucketLocationOutcome getBucketLocationOutcome = client.GetBucketLocation(getBucketLocationRequest);

        if (!getBucketLocationOutcome.IsSuccess())
            std::cout << "Error";
        else
        {
            Aws::S3::Model::BucketLocationConstraint bucketLocationConstraint =
                getBucketLocationOutcome.GetResult().GetLocationConstraint();

            Aws::String bucketRegion = Aws::S3::Model::BucketLocationConstraintMapper::GetNameForBucketLocationConstraint(bucketLocationConstraint);            
            configuration.region = bucketRegion;
        }

        // Use Transfer manager to upload stream
        auto bufferFromStream = Aws::MakeShared<UnbufferedStreamSyncBuf>("ALLOCATION_TAG", spStream, std::ios_base::in);
        auto requestStream = Aws::MakeShared<Aws::IOStream>("ALLOCATION_TAG", bufferFromStream.get());
        auto exec = Aws::MakeShared<Aws::Utils::Threading::PooledThreadExecutor>("ALLOCATION_TAG", 5);
        Aws::Transfer::TransferManagerConfiguration transferConfig(exec.get());
        transferConfig.s3Client = Aws::MakeShared<Aws::S3::S3Client>("ALLOCATION_TAG", creds, configuration, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);;
        transferConfig.computeContentMD5 = false;
        transferConfig.bufferSize = 5 * 1024 * 1024;
        transferConfig.transferBufferMaxHeapSize = transferConfig.bufferSize * 5;
        if (nTotalBytesInput < transferConfig.bufferSize)
        {
            Aws::S3::Model::PutObjectRequest putObjectRequest;
            putObjectRequest.WithBucket(BUCKET)
                .WithStorageClass(Aws::S3::Model::StorageClass::STANDARD);

            transferConfig.putObjectTemplate = putObjectRequest;
        }
        else
        {
            Aws::S3::Model::CreateMultipartUploadRequest multipartUploadRequest;
            multipartUploadRequest.WithBucket(BUCKET)
                .WithStorageClass(Aws::S3::Model::StorageClass::STANDARD);

            transferConfig.createMultipartUploadTemplate = multipartUploadRequest;
        }

        //transferConfig.transferStatusUpdatedCallback =
        //    [&](const Aws::Transfer::TransferManager*, const std::shared_ptr<const Aws::Transfer::TransferHandle>& handle)
        //{
        //    csMessage.Format(L"%s(): Transfer Status = %s", csMethod, handle->GetStatus());
        //    m_ptrStreamerObject->TraceMessage(csMessage);
        //};



        //transferConfig.uploadProgressCallback =
        //    [&](const Aws::Transfer::TransferManager*, const std::shared_ptr<const Aws::Transfer::TransferHandle>& handle)
        //{
        //    csMessage.Format(L"%s(): Upload Progress: %I64u of %I64u bytes", csMethod, handle->GetBytesTransferred(), handle->GetBytesTotalSize());
        //    m_ptrStreamerObject->TraceMessage(csMessage);
        //};

        auto transferManager = Aws::Transfer::TransferManager::Create(transferConfig);

        auto metadata = Aws::Map<Aws::String, Aws::String>();
        metadata.emplace(Aws::String(EV_METADATA_KEY), Aws::String(csMetadata));

        auto transferHandle = transferManager->UploadFile(requestStream, Aws::String(BUCKET), Aws::String(csKey),
            "application/octet-stream", metadata);

        transferHandle->WaitUntilFinished();

        auto status = transferHandle->GetStatus();

        if (status == Aws::Transfer::TransferStatus::COMPLETED)
        {
            hr = S_OK;

            size_t nParts = transferHandle->GetCompletedParts().size();
            std::cout << "Object uploaded successfully with Number of parts " << nParts << endl;
            nTotalBytesWritten = transferHandle->GetBytesTransferred();
            std::cout << "Status " << (int)status << " TotalBytesWritten " << nTotalBytesWritten << endl;
        }
        else
        {
            transferManager->AbortMultipartUpload(transferHandle);
            hr = MapAWSErrorsToHResult(transferHandle, csErrorDescription);

            //std::wcout << L"Error while uploading object. Exception " << csErrorDescription.GetString() << endl;
        }
    }
    Aws::ShutdownAPI(options);
    return hr;
}


void main(int argc, char* argv[])
{

    CStringA csFilePath = FILE_PATH_TO_UPLOAD;
    CStringA csMetadata = "EnterpriseVault File";
    CStringW csErrorDescription;

    // Open file as stream
    CComPtr<IStream> spFileStm;
    HRESULT hr = ::SHCreateStreamOnFileEx(CString(csFilePath), STGM_READWRITE | STGM_SHARE_EXCLUSIVE, FILE_ATTRIBUTE_NORMAL, FALSE, NULL, &spFileStm);
    if(FAILED(hr))
    {
        std::wcout << L"Failed to open stream on file [" << csFilePath.GetString() << L"]" << std::endl;
        exit(1);
    }
    // Get size of stream
    STATSTG statstg;
    memset(&statstg, 0, sizeof(statstg));
    hr = spFileStm->Stat(&statstg, STATFLAG_NONAME);
    if (FAILED(hr))
    {
        std::wcout << L"Failed to retrieve size of stream [" << csFilePath.GetString() << L"]" << std::endl;
        exit(1);
    }

    ULONGLONG nTotalBytesInput = statstg.cbSize.QuadPart;
    ULONGLONG nTotalBytesWritten(0);

    std::wcout << L"Size of file [" << csFilePath.GetString() << L"] " << nTotalBytesInput << L" to be uploaded" << std::endl << std::endl;

    hr = WriteObjectToStorage(spFileStm, csMetadata, nTotalBytesInput, nTotalBytesWritten, csErrorDescription);
    if (FAILED(hr))
    {
        std::wcout << csFilePath << L" uploaded failed with error " << csErrorDescription.GetString() << std::endl;
    }
    else
    {
        std::wcout << csFilePath << L" uploaded successfully." << std::endl;
    }
    spFileStm.Release();
}

---

UnbufferedStreamSyncBuf.cpp

#include "UnbufferedStreamSyncBuf.h"
#include <algorithm>

UnbufferedStreamSyncBuf::UnbufferedStreamSyncBuf(const CComPtr<IStream>& spStream, std::ios_base::openmode which)
    : std::streambuf(), pStream(spStream), bReadOnly(!(which & std::ios_base::out))
{

}

UnbufferedStreamSyncBuf::~UnbufferedStreamSyncBuf()
{
    if (!bReadOnly)
        UnbufferedStreamSyncBuf::sync();
}

bool UnbufferedStreamSyncBuf::backup()
{
    LARGE_INTEGER liMove;
    liMove.QuadPart = -1LL;
    bool bReturnVal = false;

    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);
    if (SUCCEEDED(pStream->Seek(liMove, STREAM_SEEK_CUR, NULL)))
    {
        bReturnVal = true;
    }
    return bReturnVal;
}

int UnbufferedStreamSyncBuf::overflow(int ch)
{
    int nReturnVal = traits_type::eof();
    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    if (ch != traits_type::eof())
    {
        if (SUCCEEDED(pStream->Write(&ch, 1, NULL)))
        {
            nReturnVal = ch;
        }
    }
    return nReturnVal;
}

int UnbufferedStreamSyncBuf::underflow()
{
    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    int ch = UnbufferedStreamSyncBuf::uflow();
    if (ch != traits_type::eof())
    {
        ch = backup() ? ch : traits_type::eof();
    }
    return ch;
}

int UnbufferedStreamSyncBuf::uflow()
{
    int nReturnVal = 0;
    char ch;
    ULONG cbRead;

    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    // S_FALSE indicates we've reached end of stream

    if (S_OK == pStream->Read(&ch, 1, &cbRead))
        nReturnVal = ch;
    else
        nReturnVal = traits_type::eof();


    return nReturnVal;
}

int UnbufferedStreamSyncBuf::pbackfail(int ch)
{
    if (ch != traits_type::eof())
    {
        ch = backup() ? ch : traits_type::eof();
    }
    return ch;
}

int UnbufferedStreamSyncBuf::sync()
{
    int nReturnVal = 0;

    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    if (!SUCCEEDED(pStream->Commit(STGC_DEFAULT)))
        nReturnVal = -1;

    return nReturnVal;
}

std::ios::streampos UnbufferedStreamSyncBuf::seekpos(std::ios::streampos sp, std::ios_base::openmode)
{
    std::ios::streampos posToReturn = -1;
    LARGE_INTEGER liMove;
    liMove.QuadPart = sp;

    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    if (SUCCEEDED(pStream->Seek(liMove, STREAM_SEEK_SET, NULL)))
        posToReturn = sp;

    return posToReturn;
}

std::streampos UnbufferedStreamSyncBuf::seekoff(std::streamoff off, std::ios_base::seekdir way, std::ios_base::openmode)
{
    std::ios::streampos posToReturn = -1;
    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    STREAM_SEEK sk;
    switch (way)
    {
    case std::ios_base::beg: sk = STREAM_SEEK_SET; break;
    case std::ios_base::cur: sk = STREAM_SEEK_CUR; break;
    case std::ios_base::end: sk = STREAM_SEEK_END; break;
    default: return -1;
    }
    LARGE_INTEGER liMove;
    liMove.QuadPart = static_cast<LONGLONG>(off);
    ULARGE_INTEGER uliNewPos;

    if (SUCCEEDED(pStream->Seek(liMove, sk, &uliNewPos)))
    {
        posToReturn = static_cast<std::streampos>(uliNewPos.QuadPart);
    }

    return posToReturn;
}

std::streamsize UnbufferedStreamSyncBuf::xsgetn(char *s, std::streamsize n)
{
    std::streamsize sizeToReturn = 0;

    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    ULONG cbRead;
    if (SUCCEEDED(pStream->Read(s, static_cast<ULONG>(n), &cbRead)))
    {
        sizeToReturn = static_cast<std::streamsize>(cbRead);
    }

    return sizeToReturn;
}

std::streamsize UnbufferedStreamSyncBuf::xsputn(const char *s, std::streamsize n)
{
    std::streamsize sizeToReturn = 0;
    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    ULONG cbWritten;

    if (SUCCEEDED(pStream->Write(s, static_cast<ULONG>(n), &cbWritten)))
    {
        sizeToReturn = static_cast<std::streamsize>(cbWritten);
    }

    return sizeToReturn;
}

std::streamsize UnbufferedStreamSyncBuf::showmanyc()
{
    std::streamsize sizeToReturn = 0;
    CComCritSecLock<CComAutoCriticalSection> lock(m_CS);

    STATSTG stat;
    if (SUCCEEDED(pStream->Stat(&stat, STATFLAG_NONAME)))
    {
        std::streampos lastPos = static_cast<std::streampos>(stat.cbSize.QuadPart - 1);
        LARGE_INTEGER liMove;
        liMove.QuadPart = 0LL;
        ULARGE_INTEGER uliNewPos;
        if (SUCCEEDED(pStream->Seek(liMove, STREAM_SEEK_CUR, &uliNewPos)))
        {
            sizeToReturn = std::max<std::streamsize>(lastPos - static_cast<std::streampos>(uliNewPos.QuadPart), 0);
        }
    }

    return sizeToReturn;
}

---

UnbufferedStreamSyncBuf.h

#pragma once
#include <ios>
#include <atlstr.h>
#include "atlbase.h"
using namespace ATL;

// Read-write unbuffered streambuf implementation which uses no
// internal buffers (adapted Matt Austern's example from http://www.drdobbs.com/184401305)
class UnbufferedStreamSyncBuf : public std::streambuf
{
public:
    UnbufferedStreamSyncBuf(const CComPtr<IStream>& spStream, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out);
    ~UnbufferedStreamSyncBuf();

protected:
    virtual int overflow(int ch = traits_type::eof());
    virtual int underflow();
    virtual int uflow();
    virtual int pbackfail(int ch = traits_type::eof());
    virtual int sync();
    virtual std::streampos seekpos(std::streampos sp, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out);
    virtual std::streampos seekoff(std::streamoff off, std::ios_base::seekdir way, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out);
    virtual std::streamsize xsgetn(char *s, std::streamsize n);
    virtual std::streamsize xsputn(const char *s, std::streamsize n);
    virtual std::streamsize showmanyc();

private:
    CComPtr<IStream> pStream;
    CComAutoCriticalSection m_CS;
    bool bReadOnly;
    bool backup();
};