Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[S3] PutObject High Memory Usage #894

Closed
indy-singh opened this issue Mar 7, 2018 · 5 comments
Closed

[S3] PutObject High Memory Usage #894

indy-singh opened this issue Mar 7, 2018 · 5 comments

Comments

@indy-singh
Copy link
Contributor

indy-singh commented Mar 7, 2018

Whilst under taking performance work centred around memory, we have discovered that this method:-

public virtual PutObjectResponse PutObject(PutObjectRequest request)

Causes LOH allocations as well as fragmentation. As we are uploading small sized images (many thousand times a day) we can't see why this should be the case. This was revealed by taking dumps of production servers and dumping out the byte arrays. The byte arrays in question contained ;chunk-signature= with image data afterwards.

Using dotTrace we have been able to narrow down the allocations to ChunkedUploadWrapperStream specifically the constructor:-

internal ChunkedUploadWrapperStream(Stream stream, int wrappedStreamBufferSize, AWS4SigningResult headerSigningResult)

For every invocation of the constructor two byte arrays are created each having a minimum size of 131072:-

public static readonly int DefaultChunkSize = 128*1024;

The threshold for objects to go onto the LOH is 85,000 bytes. We believe this is the root cause of the problem we are experiencing.

Expected Behavior

That uploading small files doesn't cause LOH allocation and further on down the line fragmentation.

Current Behavior

Every time you upload an image to S3 two byte arrays are created with a minimum size of 131072. This is greater than the 85,000 threshold causing both byte arrays to be allocated directly to the LOH.

Possible Solution

There are a few possible solutions:-

  1. Use System.Buffers to have a pool of byte arrays that can be recycled.
  2. Use Microsoft.IO.RecyclableMemoryStream to have a pool of MemoryStreams - we have used this to great effect in our own code to reduce LOH allocations by 99.6%.
  3. Expose DefaultChunkSize so that consumers of the API can set it themselves
  4. Lower DefaultChunkSize to a number that is by default lower than the LOH threshold. Microsoft use 81920 in Stream:- https://referencesource.microsoft.com/#mscorlib/system/io/stream.cs,50

Steps to Reproduce (for bugs)

using System;
using System.IO;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;

namespace AWS_S3_Test_Harness
{
	public class Program
	{
		private const string _bucketName = "_bucketName";
		private const string _accessKey = "_accessKey";
		private const string _secretKey = "_secretKey";
		private static readonly BasicAWSCredentials _credentials = new BasicAWSCredentials(_accessKey, _secretKey);

		public static void Main(string[] args)
		{
			var readAllBytes = File.ReadAllBytes(@"..\..\sample_image.png"); // https://a0.awsstatic.com/main/images/logos/aws_logo_smile_1200x630.png

			for (int index = 0; index < 200; index++)
			{
				PutObject(index + ".png", new MemoryStream(readAllBytes));   
			}
		}

		private static void PutObject(string fileName, Stream fileContent)
		{
			try
			{
				using(var client = new AmazonS3Client(_credentials, RegionEndpoint.EUWest1))
				{
					var putObjectRequest = new PutObjectRequest
					{
						Timeout = TimeSpan.FromSeconds(5),
						BucketName = _bucketName,
						CannedACL = S3CannedACL.PublicRead,
						InputStream = fileContent,
						StorageClass = S3StorageClass.Standard,
						ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
						Key = "aws-s3-test-harness/" + fileName
					};

					client.PutObject(putObjectRequest);
				}
			}
			catch(Exception exception)
			{
				Console.WriteLine(exception);
			}
		}
	}
}

Context

Our platform provides an API in which our customers can send us images. We then transform and upload these images to S3. We receive many thousands of images per hour.

Your Environment

  • AWSSDK.Core version used: 3.3.21.15
  • Service assembly and version used: 3.3.17.1
  • Operating System and version: Windows Server 2012 R2 6.3 Build 9600
  • Visual Studio version: 2015 Professional Update 3
  • Targeted .NET platform: 4.5.2
@indy-singh
Copy link
Contributor Author

indy-singh commented Mar 10, 2018

I realise I didn't add any statistics to my opening post. The test harness in the opening post uses this image and just uploads it 200 times via:-

public virtual PutObjectResponse PutObject(PutObjectRequest request)

Total .NET Allocations: 114MB
SOH Allocations: 64MB
LOH Allocations: 50MB

all_calls

And then if we drill down into one call to PutObject:-

single_call

It looks like every call to PutObject has a fixed cost of 0.3 MB. I'm happy to submit to PR that lowers DefaultChunkSize to 81920 if it is welcome.

Thanks,
Indy

indy-singh added a commit to indy-singh/aws-sdk-net that referenced this issue Mar 10, 2018
See aws#894 for detailed description
@indy-singh
Copy link
Contributor Author

indy-singh commented Mar 11, 2018

I made the change (indy-singh@008c5ff) locally and complied the required projects in release and re-ran my test harness:-

Total .NET Allocations: 98MB
SOH Allocations: 98MB
LOH Allocations: 0.4MB

all_calls

And then if we drill down into one call to PutObject:-

single_call

Thanks,
Indy

@indy-singh indy-singh changed the title [S3] PutObject Large Object Heap Allocations and Fragmentation [S3] PutObject High Memory Usage Mar 14, 2018
@indy-singh
Copy link
Contributor Author

indy-singh commented Apr 2, 2018

This bug was fixed and released in AWSSDK.Core 3.3.21.19

Core Version Total RAM Small Object Heap Large Object Heap
v3.3.21.18 116 MB 66 MB 50 MB
v3.3.21.19 99 MB 98 MB 0.4 MB

Incidentally, GetPreSignedURL offers much better performance memory wise:-

public string GetPreSignedURL(GetPreSignedUrlRequest request)

Total RAM Small Object Heap Large Object Heap
32 MB 32 MB 0.4 MB

Thanks for fixing this bug. Closing.

Cheers,
Indy

@rsrini83
Copy link

rsrini83 commented Sep 8, 2018

HI Indy-Singh,
We are observing the similar behavior(99% RAM usage) when uploading 100+ GB file using TransferUtility. We are using AWS Core 3.3.8 version(little old). However i just want to understand is it the same issue or am i facing different issue ?

System Configuration;
OS: Windows 2012 R2
RAM: 16 GB
Usage: 15.6 GB
AWS Core: 3.3.8

--Srinu

@indy-singh
Copy link
Contributor Author

indy-singh commented Sep 12, 2018

@rsrini83 I would update to the latest version of AWS Core and try again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants