S3 - OpenStreamAsync/GetObjectAsync randomly end with "The response ended prematurely with at least X bytes expected"

[Rayzbam]

Describe the bug

When we upload big file ( > 200Mb in our usecase ) we, randomly, have an issue if we try to get the stream right after upload it on the S3 Storage

We're using the TransferUtility methods as follow :

async Task UploadAsync(AmazonS3Client client, string bucket, string storageName, Stream documentStream)
{
    var transferUtility = new TransferUtility(client);
    var request = new TransferUtilityUploadRequest { AutoCloseStream = false, BucketName = bucket, Key = storageName, InputStream = documentStream };
    var lastPercentage = 0;
    request.UploadProgressEvent += (s, e) => Request_UploadProgressEvent(s, e, ref lastPercentage);
    await transferUtility.UploadAsync(request, CancellationToken.None);
}

async Task<Stream> GetStreamAsync(AmazonS3Client client, string bucket, string storageName)
{
    var metadata = await client.GetObjectMetadataAsync(bucket, storageName);
    if (metadata.PartsCount.HasValue)
        Console.WriteLine($"Part count is {metadata.PartsCount}");

    var transferUtility = new TransferUtility(client);
    var stream = await transferUtility.OpenStreamAsync(bucket, storageName, CancellationToken.None);
    return stream;
}

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I want to be able to work with the stream of a blob i just uploaded without getting any exception.

Current Behavior

We encountered different exceptions according to the operations we're doing with the stream.
The first one is a simple CopyToAsync to a local MemoryStream

System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, Boolean async, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in _localRepo__\Program.cs:line 44
   at Program.<Main>(String[] args)

The second one is clearly the same but we're trying to broadcast the stream directly to an Azure Storage :

The response ended prematurely, with at least 97892332 additional bytes expected. (ResponseEnded)

And if i'm not using a custom HTTPHandlerFactory

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, Boolean async, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)

Reproduction Steps

My only usecase is in a loop for now, but it seems to encounter the issue pretty quickly on my side (less than 30 iterations, sometimes even the 2nd iteration is throwing) :

var s3config = new AmazonS3Config() { ServiceURL = "__S3URL__", ForcePathStyle = true };
var s3Client = new AmazonS3Client(accessKeyId, accessSecretKey, s3config);
string bucketName = "__BUCKETNAME__";

for (int i = 0; i < 1000; i++)
{
    try
    {
        Console.WriteLine($"Index : {i} started at {DateTime.Now.ToString("HH:mm.ss")}");

        using (var stream = new FileStream("bigFile.xml", FileMode.Open))
            await UploadAsync(s3Client, bucketName, storageName, stream);

        Console.WriteLine($"S3 put. Start reading stream to copy at {DateTime.Now.ToString("HH:mm.ss")}");

        using (var ms = new MemoryStream())
        using (var inStream = await GetStreamAsync(s3Client, bucketName, storageName))
            await inStream.CopyToAsync(ms);

        Console.WriteLine($"Stream retrieved at {DateTime.Now.ToString("HH:mm.ss")}");
        await s3Client.DeleteObjectAsync(bucketName, storageName);
        Console.WriteLine($"Index : {i} ended at {DateTime.Now.ToString("HH:mm.ss")}");
    }
    catch (Exception)
    {
        Console.ForegroundColor = ConsoleColor.Red;
        Console.WriteLine($"Failed at {DateTime.Now.ToString("HH:mm.ss")}");
        Console.ResetColor();
        throw;
    }
}

Possible Solution

No response

Additional Information/Context

Only having the issue with big files, the test provided was run with a 220Mb file. Having the issue with a 50Mb file too.

AWS .NET SDK and/or Package version used

AWSSDK.S3 v3.7.405.7

Targeted .NET Platform

.NET 8

Operating System and version

Windows 11

[dscpinheiro]

In your reproduction code, you're setting the ServiceURL. Are you using the actual S3 service or a 3rd-party implementation?

[Rayzbam]

We are using a 3rd-party implementation provided by Nutanix called Nutanix objects
We already opened an issue on their side too.
I just can’t be sure the issue is on their implementation.

[dscpinheiro]

Got it. FYI I did try your example against S3 (using a smaller object - around 70 MB), and it did not fail once for the 1000 iterations.

We also made a change to the SDK last month (#3610) and some 3rd-party implementations were impacted, but all reports we received were related to uploading objects (not downloading them).

[Rayzbam]

Thanks for your time on my repro.
Tbh i opened this as a « bug » because i don’t have any other good one to post, just wanted to be sure my use case is a valid one and not doing something wrong with the SDK.
I will try tomorrow on a « classic » S3 and let you know the results.
70Mb seems to reproduce the issue but, clearly, the 200Mb+ seems to expose this issue more precisely (on my 3rd-party implementation of course).
The main difference between an on premise implementation and an AWS S3 for us is probably the network interface between the client and the server.
I am looking to something which can help my network team or the one of the 3rd party implementation to localize the source of the issue.
Do you have any advice ?

[dscpinheiro]

The exceptions you're seeing make me believe it's an issue with the (3rd-party) server not being able to handle the load, however if you're able to reproduce it with S3 we can reach out to the service team for help.

This won't solve the problem, but you can also enable debug logging by adding these lines prior to creating the S3 client (I haven't tested with that specific 3rd-party, but with S3 this will show the canonical request, request ID, and other useful troubleshooting information):

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

[Rayzbam]

Seems a good one.
Will try this and let you know.
I am not closing the issue for now if you don’t mind.
It’s a big issue in my company and i would like to keep every possibility open for now.

[Rayzbam]

Clearly, these lines have added a lot of information.
The last log lines i have before reproducing my issue are pretty "normal" imo.
We can see that the GET made to the S3 is coming back with the content length i expect (220Mo), and during the socket reading process, it seems that socket is closed by the remote host.

We tried on a AWS S3 and, obviously, it worked like a charm.
Unless these logs can be useful to someone, It seems to be an issue on our side, probably a network issue, or a mistake in the S3 3rd party implementation (maybe ?)