(104): Connection reset by peer

[wahmedswl]

Describe the bug

We are seeing internment failures on our Network Calls (Http + SOAP/WCF) on .Net Core 3.1. The invoked endpoints/services work fine and throws sometimes as well. We are getting following error

System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection: Connection reset by peer.)
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer

Expected Behavior

We shouldn't see this error and network calls should succeed without issue.

Current Behavior

System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection: Connection reset by peer.)
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer

Reproduction Steps

We are getting these inside Lambda on Network Calls. These are invoked via Step Function at Hourly interval and around 50/60+ API calls at each run.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

<PackageReference Include="AWSSDK.Core" Version="3.7.2.2" />

Targeted .NET Platform

.Net Core 3.1

Operating System and version

Lambda

[ashishdhingra]

Hi @wahmedswl,

Good morning.

Thanks for reporting the issue. Could you please share the following to reproduce the issue:

1. Sample code solution to reproduce the issue.
2. Steps to deploy the code solution with any specific settings.
3. The error that you shared doesn't demonstrate that the exception is thrown by the AWS SDK. What is the specific call that you are making when you get this error? Are you making network call to specific host which might not be whitelisted?
4. Enable verbose logging using the code below and share the detailed logs:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

6. Some of the online articles/issues, e.g., Connection reset by peer when using AWS Lambda npgsql/npgsql#3559, mention root cause as the remote entity reset the connection due to connection timeout, others point out issue due to higher concurrency execution.

• You might want to check any timeout setting that might cause this error.
• Concurrency setting in Lambda (refer Configuring reserved concurrency)

Thanks,
Ashish

[normj]

Have you been able try deploying the function using .NET 6? .NET Core 3.1 is out of support from Microsoft and this week entered the first phase of deprecation on Lambda.

Is it always the same AWS service throwing this error? Are you creating the service client as part of every invoke or is it shared across invokes?

[wahmedswl]

@normj Nope, there are multiple api endpoints. Initially we thought that it's 3rd party vendor but later we saw happening on multiple vendors. That's why we raised this issue.
@ashishdhingra Yes, this is not related to AWS Sdk but 3rd party api invoke via Lambda.

[normj]

My guess is the third party library is maintaining socket level connection and the library/third party service doesn't like when Lambda freezes the compute container between invokes. Maybe the service is expecting keep alive messages to be sent while function is frozen. Is it possible to create the third party client for each invoke or catch the error and recreate the client if you get this error?

[wahmedswl]

@normj Client is not cached/kept and created on every network call.

[ashishdhingra]

@wahmedswl Please help share the information mentioned in #1474 (comment) to investigate further.

Thanks,
Ashish

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.