Whilst under taking performance work centred around memory, we have discovered that this method:-
|
public virtual PutObjectResponse PutObject(PutObjectRequest request) |
Causes LOH allocations as well as fragmentation. As we are uploading small sized images (many thousand times a day) we can't see why this should be the case. This was revealed by taking dumps of production servers and dumping out the byte arrays. The byte arrays in question contained ;chunk-signature= with image data afterwards.
Using dotTrace we have been able to narrow down the allocations to ChunkedUploadWrapperStream specifically the constructor:-
|
internal ChunkedUploadWrapperStream(Stream stream, int wrappedStreamBufferSize, AWS4SigningResult headerSigningResult) |
For every invocation of the constructor two byte arrays are created each having a minimum size of 131072:-
|
public static readonly int DefaultChunkSize = 128*1024; |
The threshold for objects to go onto the LOH is 85,000 bytes. We believe this is the root cause of the problem we are experiencing.
Expected Behavior
That uploading small files doesn't cause LOH allocation and further on down the line fragmentation.
Current Behavior
Every time you upload an image to S3 two byte arrays are created with a minimum size of 131072. This is greater than the 85,000 threshold causing both byte arrays to be allocated directly to the LOH.
Possible Solution
There are a few possible solutions:-
- Use System.Buffers to have a pool of byte arrays that can be recycled.
- Use Microsoft.IO.RecyclableMemoryStream to have a pool of MemoryStreams - we have used this to great effect in our own code to reduce LOH allocations by 99.6%.
- Expose
DefaultChunkSize so that consumers of the API can set it themselves
- Lower
DefaultChunkSize to a number that is by default lower than the LOH threshold. Microsoft use 81920 in Stream:- https://referencesource.microsoft.com/#mscorlib/system/io/stream.cs,50
Steps to Reproduce (for bugs)
using System;
using System.IO;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;
namespace AWS_S3_Test_Harness
{
public class Program
{
private const string _bucketName = "_bucketName";
private const string _accessKey = "_accessKey";
private const string _secretKey = "_secretKey";
private static readonly BasicAWSCredentials _credentials = new BasicAWSCredentials(_accessKey, _secretKey);
public static void Main(string[] args)
{
var readAllBytes = File.ReadAllBytes(@"..\..\sample_image.png"); // https://a0.awsstatic.com/main/images/logos/aws_logo_smile_1200x630.png
for (int index = 0; index < 200; index++)
{
PutObject(index + ".png", new MemoryStream(readAllBytes));
}
}
private static void PutObject(string fileName, Stream fileContent)
{
try
{
using(var client = new AmazonS3Client(_credentials, RegionEndpoint.EUWest1))
{
var putObjectRequest = new PutObjectRequest
{
Timeout = TimeSpan.FromSeconds(5),
BucketName = _bucketName,
CannedACL = S3CannedACL.PublicRead,
InputStream = fileContent,
StorageClass = S3StorageClass.Standard,
ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
Key = "aws-s3-test-harness/" + fileName
};
client.PutObject(putObjectRequest);
}
}
catch(Exception exception)
{
Console.WriteLine(exception);
}
}
}
}
Context
Our platform provides an API in which our customers can send us images. We then transform and upload these images to S3. We receive many thousands of images per hour.
Your Environment
- AWSSDK.Core version used: 3.3.21.15
- Service assembly and version used: 3.3.17.1
- Operating System and version: Windows Server 2012 R2 6.3 Build 9600
- Visual Studio version: 2015 Professional Update 3
- Targeted .NET platform: 4.5.2
Whilst under taking performance work centred around memory, we have discovered that this method:-
aws-sdk-net/sdk/src/Services/S3/Generated/_bcl45/AmazonS3Client.cs
Line 4421 in 9814e24
Causes LOH allocations as well as fragmentation. As we are uploading small sized images (many thousand times a day) we can't see why this should be the case. This was revealed by taking dumps of production servers and dumping out the byte arrays. The byte arrays in question contained
;chunk-signature=with image data afterwards.Using dotTrace we have been able to narrow down the allocations to
ChunkedUploadWrapperStreamspecifically the constructor:-aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/ChunkedUploadWrapperStream.cs
Line 73 in 9814e24
For every invocation of the constructor two byte arrays are created each having a minimum size of
131072:-aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/ChunkedUploadWrapperStream.cs
Line 40 in 9814e24
The threshold for objects to go onto the LOH is 85,000 bytes. We believe this is the root cause of the problem we are experiencing.
Expected Behavior
That uploading small files doesn't cause LOH allocation and further on down the line fragmentation.
Current Behavior
Every time you upload an image to S3 two byte arrays are created with a minimum size of 131072. This is greater than the 85,000 threshold causing both byte arrays to be allocated directly to the LOH.
Possible Solution
There are a few possible solutions:-
DefaultChunkSizeso that consumers of the API can set it themselvesDefaultChunkSizeto a number that is by default lower than the LOH threshold. Microsoft use 81920 inStream:- https://referencesource.microsoft.com/#mscorlib/system/io/stream.cs,50Steps to Reproduce (for bugs)
Context
Our platform provides an API in which our customers can send us images. We then transform and upload these images to S3. We receive many thousands of images per hour.
Your Environment