Automatic retry on IOException/SocketException

[bill-poole]

The default/configurable retry strategies do not seem to support retries on an IOException. However, the AWS SDK does consider an IOException to be transient under some conditions. I find that occasionally I receive an IOException caused by an underlying SocketException with error code 10054 (ConnectionReset), which means "an existing connection was forcibly closed by the remote host."

The AWS .NET SDK logic pertaining to retrying on IOException is (from RetryPolicy.cs):

// An IOException was thrown by the underlying http client.
// FileNotFoundException is not considered a transient error because
// we don't consider local .NET assembly file changes to be happening.
// If a FileNotFoundException happens there is most likey a bad install
// of the SDK or .NET assembly binding issue.
if (exception is IOException && exception is not FileNotFoundException)
{

#if !NETSTANDARD  // ThreadAbortException is not NetStandard

    // Don't retry IOExceptions that are caused by a ThreadAbortException
    if (ExceptionUtils.IsInnerException<ThreadAbortException>(exception))
        return false;

#endif
    // Retry all other IOExceptions
    return true;
}
else if (ExceptionUtils.IsInnerException<IOException>(exception))
{
    return true;
}       

It would be very useful if this library would allow configuration of and provide a default for a retry policy for these transient exceptions.

The relevant excerpt of my stack trace is:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
       ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
         at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
         at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at EfficientDynamoDb.Internal.HttpApi.SendAsync(DynamoDbContextConfig config, HttpContent httpContent, CancellationToken cancellationToken)
         at EfficientDynamoDb.DynamoDbContext.GetItemAsync[TEntity](DdbClassInfo classInfo, BuilderNode node, CancellationToken cancellationToken)
         at EfficientDynamoDb.Operations.GetItem.GetItemEntityRequestBuilder`1.ToItemAsync(CancellationToken cancellationToken)

[bill-poole]

I also get the following exception occasionally, which also seems to be a transient exception that should be automatically retried.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
       ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
         at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at EfficientDynamoDb.Internal.HttpApi.SendAsync(DynamoDbContextConfig config, HttpContent httpContent, CancellationToken cancellationToken)
         at EfficientDynamoDb.DynamoDbContext.GetItemAsync[TEntity](DdbClassInfo classInfo, BuilderNode node, CancellationToken cancellationToken)
         at EfficientDynamoDb.Operations.GetItem.GetItemEntityRequestBuilder`1.ToItemAsync(CancellationToken cancellationToken)

[jakejscott]

We have also seen this happening. @firenero Do you mind if I have a go at sending a PR for this?

System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Memory`1 socketAddress, SocketFlags receivedFlags, SocketError socketError)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---

--- End of inner exception stack trace ---
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at EfficientDynamoDb.Internal.HttpApi.SendAsync(DynamoDbContextConfig config, HttpContent httpContent, CancellationToken cancellationToken)
   at EfficientDynamoDb.DynamoDbContext.PutItemAsync[TEntity](BuilderNode node, CancellationToken cancellationToken)
   at EfficientDynamoDb.Operations.PutItem.PutItemEntityRequestBuilder`1.ExecuteAsync(CancellationToken cancellationToken)

[jakejscott]

I think a workaround is to implement our own IHttpClientFactory, something like this

using System.Net;
using System.Net.Sockets;
using EfficientDynamoDb.Configs.Http;

public class DynamoDbHttpClientFactory : IHttpClientFactory
{
    public static readonly DynamoDbHttpClientFactory Instance = new();

    private readonly HttpClient _httpClient;

    private DynamoDbHttpClientFactory()
    {
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.GZip
        };

        var resilienceHandler = new HttpResilienceHandler(handler);

        _httpClient = new HttpClient(resilienceHandler);
    }

    public HttpClient CreateHttpClient() => _httpClient;

    private class HttpResilienceHandler(HttpMessageHandler innerHandler) : DelegatingHandler(innerHandler)
    {
        private const int MaxRetries = 10;

        private static readonly DecorrelatedJitter _jitter = new(baseDelayMs: 100, maxDelayMs: 30_000);

        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
        {
            for (var attempt = 0; attempt <= MaxRetries; attempt++)
            {
                try
                {
                    return await base.SendAsync(request, ct);
                }
                catch (HttpRequestException e) when (e.InnerException is IOException or HttpIOException or SocketException)
                {
                    var delay = _jitter.CalculateDelay(attempt);
                    await Task.Delay(delay, ct);
                }
            }

            return await base.SendAsync(request, ct);
        }
    }
}

[firenero]

@jakejscott sorry for the delay. Yes, I'm open to merging a PR to fix this one. Your #276 should work but you closed it for some reason. Overall, I think it's better to implement retries for HttpClient directly like in your last example. Are you still willing to submit a PR for that?

[jakejscott]

Hey @firenero awesome yeah I'll send a pr

[jakejscott]

@firenero I'll try out 0.9.18-alpha.0.8 and let you know how it goes.

[jakejscott]

@firenero This has fixed our issue. Not seeing any of these error's since we deployed a week ago! Any chance you could publish 0.9.18? We'd love to upgrade all our services so that they are not on the alpha version. Thanks!

[firenero]

Released v0.9.18. Thanks for reporting this issue and submitting a PR!

[jakejscott]

Thanks heaps @firenero and @bill-poole, glad to get this one sorted!