[S3] PutObject High Memory Usage

[indy-singh]

Whilst under taking performance work centred around memory, we have discovered that this method:-

aws-sdk-net/sdk/src/Services/S3/Generated/_bcl45/AmazonS3Client.cs

Line 4421 in 9814e24

public virtual PutObjectResponse PutObject(PutObjectRequest request)

Causes LOH allocations as well as fragmentation. As we are uploading small sized images (many thousand times a day) we can't see why this should be the case. This was revealed by taking dumps of production servers and dumping out the byte arrays. The byte arrays in question contained ;chunk-signature= with image data afterwards.

Using dotTrace we have been able to narrow down the allocations to ChunkedUploadWrapperStream specifically the constructor:-

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/ChunkedUploadWrapperStream.cs

Line 73 in 9814e24

internal ChunkedUploadWrapperStream(Stream stream, int wrappedStreamBufferSize, AWS4SigningResult headerSigningResult)

For every invocation of the constructor two byte arrays are created each having a minimum size of 131072:-

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/ChunkedUploadWrapperStream.cs

Line 40 in 9814e24

public static readonly int DefaultChunkSize = 128*1024;

The threshold for objects to go onto the LOH is 85,000 bytes. We believe this is the root cause of the problem we are experiencing.

Expected Behavior

That uploading small files doesn't cause LOH allocation and further on down the line fragmentation.

Current Behavior

Every time you upload an image to S3 two byte arrays are created with a minimum size of 131072. This is greater than the 85,000 threshold causing both byte arrays to be allocated directly to the LOH.

Possible Solution

There are a few possible solutions:-

1. Use System.Buffers to have a pool of byte arrays that can be recycled.
2. Use Microsoft.IO.RecyclableMemoryStream to have a pool of MemoryStreams - we have used this to great effect in our own code to reduce LOH allocations by 99.6%.
3. Expose DefaultChunkSize so that consumers of the API can set it themselves
4. Lower DefaultChunkSize to a number that is by default lower than the LOH threshold. Microsoft use 81920 in Stream:- https://referencesource.microsoft.com/#mscorlib/system/io/stream.cs,50

Steps to Reproduce (for bugs)

using System;
using System.IO;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;

namespace AWS_S3_Test_Harness
{
	public class Program
	{
		private const string _bucketName = "_bucketName";
		private const string _accessKey = "_accessKey";
		private const string _secretKey = "_secretKey";
		private static readonly BasicAWSCredentials _credentials = new BasicAWSCredentials(_accessKey, _secretKey);

		public static void Main(string[] args)
		{
			var readAllBytes = File.ReadAllBytes(@"..\..\sample_image.png"); // https://a0.awsstatic.com/main/images/logos/aws_logo_smile_1200x630.png

			for (int index = 0; index < 200; index++)
			{
				PutObject(index + ".png", new MemoryStream(readAllBytes));   
			}
		}

		private static void PutObject(string fileName, Stream fileContent)
		{
			try
			{
				using(var client = new AmazonS3Client(_credentials, RegionEndpoint.EUWest1))
				{
					var putObjectRequest = new PutObjectRequest
					{
						Timeout = TimeSpan.FromSeconds(5),
						BucketName = _bucketName,
						CannedACL = S3CannedACL.PublicRead,
						InputStream = fileContent,
						StorageClass = S3StorageClass.Standard,
						ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
						Key = "aws-s3-test-harness/" + fileName
					};

					client.PutObject(putObjectRequest);
				}
			}
			catch(Exception exception)
			{
				Console.WriteLine(exception);
			}
		}
	}
}

Context

Our platform provides an API in which our customers can send us images. We then transform and upload these images to S3. We receive many thousands of images per hour.

Your Environment

• AWSSDK.Core version used: 3.3.21.15
• Service assembly and version used: 3.3.17.1
• Operating System and version: Windows Server 2012 R2 6.3 Build 9600
• Visual Studio version: 2015 Professional Update 3
• Targeted .NET platform: 4.5.2

[indy-singh]

I realise I didn't add any statistics to my opening post. The test harness in the opening post uses this image and just uploads it 200 times via:-

aws-sdk-net/sdk/src/Services/S3/Generated/_bcl45/AmazonS3Client.cs

Line 4421 in 9814e24

public virtual PutObjectResponse PutObject(PutObjectRequest request)

Total .NET Allocations: 114MB
SOH Allocations: 64MB
LOH Allocations: 50MB

And then if we drill down into one call to PutObject:-

It looks like every call to PutObject has a fixed cost of 0.3 MB. I'm happy to submit to PR that lowers DefaultChunkSize to 81920 if it is welcome.

Thanks,
Indy

[indy-singh]

I made the change (indy-singh@008c5ff) locally and complied the required projects in release and re-ran my test harness:-

Total .NET Allocations: 98MB
SOH Allocations: 98MB
LOH Allocations: 0.4MB

And then if we drill down into one call to PutObject:-

Thanks,
Indy

[indy-singh]

This bug was fixed and released in AWSSDK.Core 3.3.21.19

Core Version	Total RAM	Small Object Heap	Large Object Heap
v3.3.21.18	116 MB	66 MB	50 MB
v3.3.21.19	99 MB	98 MB	0.4 MB

Incidentally, GetPreSignedURL offers much better performance memory wise:-

aws-sdk-net/sdk/src/Services/S3/Custom/_bcl+coreclr+pcl/AmazonS3Client.Extensions.cs

Line 81 in 37b2546

public string GetPreSignedURL(GetPreSignedUrlRequest request)

Total RAM	Small Object Heap	Large Object Heap
32 MB	32 MB	0.4 MB

Thanks for fixing this bug. Closing.

Cheers,
Indy

[rsrini83]

HI Indy-Singh,
We are observing the similar behavior(99% RAM usage) when uploading 100+ GB file using TransferUtility. We are using AWS Core 3.3.8 version(little old). However i just want to understand is it the same issue or am i facing different issue ?

System Configuration;
OS: Windows 2012 R2
RAM: 16 GB
Usage: 15.6 GB
AWS Core: 3.3.8

--Srinu

[indy-singh]

@rsrini83 I would update to the latest version of AWS Core and try again.