Only 2 requests are working with ASMX client

[Bykiev]

I've generated an ASMX service client in Visual Studio, then in a loop I'm creating the new client instance, calling the method and closing the client. On 3 try I'm getting error:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

I've reproduced the issue with .NET6, .NET8. I've tried the same with .NET Framework 4.7.2 but it's working fine without any errors. If I'll add about 5 sec delay between the requests it seems to be working fine

I can't provide access to third-party ASMX service, but how can I trace the root of the issue?

Microsoft.Tools.ServiceModel.Svcutil 2.1.0

The code is pretty simple:

var binding = new BasicHttpBinding(BasicHttpSecurityMode.Transport);
binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;
binding.AllowCookies = true;
binding.MaxBufferSize = int.MaxValue;
binding.MaxReceivedMessageSize = int.MaxValue;
binding.ReaderQuotas = System.Xml.XmlDictionaryReaderQuotas.Max;

for (int i = 0; i < 10; i++)
{
  using (var svc = new ServiceSoapClient(binding, new EndpointAddress(new Uri(url))))
  {
      svc.ClientCredentials.UserName.UserName = login;
      svc.ClientCredentials.UserName.Password = password;
  
   // call methods
  }
}

StackTraces:

   in System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   in System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   in System.Net.Sockets.Socket.<<ConnectAsync>g__WaitForConnectWithCancellation|285_0>d.MoveNext()
   in System.Net.Http.HttpConnectionPool.<ConnectToTcpHostAsync>d__104.MoveNext()

   in System.Net.Http.HttpConnectionPool.<ConnectToTcpHostAsync>d__104.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.HttpConnectionPool.<ConnectAsync>d__103.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.HttpConnectionPool.<CreateHttp11ConnectionAsync>d__105.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.HttpConnectionPool.<AddHttp11ConnectionAsync>d__79.MoveNext()
   in System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.<WaitWithCancellationAsync>d__1.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.<WaitForConnectionWithTelemetryAsync>d__6.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__89.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.AuthenticationHelper.<SendWithAuthAsync>d__17.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   in System.Threading.Tasks.ValueTask`1.get_Result()
   in System.Net.Http.DecompressionHandler.<SendAsync>d__16.MoveNext()
   in System.Net.Http.HttpClient.<<SendAsync>g__Core|83_0>d.MoveNext()
   in System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<SendRequestAsync>d__13.MoveNext()

 in System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(HttpRequestException requestException, HttpRequestMessage request, HttpAbortReason abortReason)
   in System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<SendRequestAsync>d__13.MoveNext()
   in System.ServiceModel.Channels.RequestChannel.<RequestAsync>d__33.MoveNext()
   in System.ServiceModel.Channels.RequestChannel.<RequestAsyncInternal>d__32.MoveNext()
   in System.Runtime.TaskHelpers.WaitForCompletionNoSpin[TResult](Task`1 task)
   in System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   in System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   in System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   in System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
   in System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(MethodCall methodCall, ProxyOperationRuntime operation)
   in System.ServiceModel.Channels.ServiceChannelProxy.Invoke(MethodInfo targetMethod, Object[] args)
   in generatedProxy_1.GetData(GetDataRequest )

[mconnew]

Add the following as the first statement in your code:

ServiceSoapClient.CacheSetting = CacheSetting.AlwaysOn;

Because WCF doesn't use config based configuration for .NET, it can't be presumed safe to reuse a ChannelFactory between instances of a class that derives from ClientBase<TChannel> as the binding and/or endpoint can be different between instantiations. When using config, as long as the config name was the same, we knew it was safe so the ChannelFactory was reused between instances of a generated client.

As you are using the same Binding instance (it requires Object.ReferenceEquals to return true), and the same endpoint address (here it uses .Equals so a new instance each time is fine as long as it's the same uri), setting the CacheSetting to AlwaysOn will mean it reuses the ChannelFactory.

WCF on .NET Framework uses HttpWebRequest which pools connections process wide. On .NET [Core], we use HttpClient/HttpClientHandler (HttpWebRequest in .NET is a wrapper around this and has some performance issues as it's for compat usage only) which scopes the connection pool to the HttpClientHandler instance. As each ChannelFactory creates its own instance of HttpClientHandler (it has to because credentials are set at the handler level and not the request level), the connection pool is scoped to the ChannelFactory.

I presume you are making calls against either IIS Express or IIS installed on a client version of Windows. Both of those have a limit of 10 connections (it's not a server OS). As each client creates its own ChannelFactory, which creates its own instance of HttpClientHandler, a persistent connection is maintained after each call. You basically used up all the connections your server allows. Adding the delay allows time for the connection to be cleaned up after you dispose the client, which closes the ChannelFactory.

Enabling ChannelFactory caching solves this.

[Bykiev]

@mconnew, thank you, I've tried this option, but nothing has been changed. I'm testing with a console applications. The native error code is 10060

[mconnew]

Try using netstat to see how many sockets are open.

[Bykiev]

Sorry, do I understand you correctly that I should execute netstat -ano command?

[Bykiev]

Sysinternals TCPView shows Listening 122, Time Wait 97, Close Wait: 20

[Bykiev]

.NET framework 4.7.2 requests stat:

time elapsed request time (sec)
09:00:06.37:  elapsed: 0,8013172
09:00:06.60:  elapsed: 0,2210235
09:00:06.83:  elapsed: 0,2351614
09:00:07.08:  elapsed: 0,2519398
09:00:07.53:  elapsed: 0,4506108
09:00:07.79:  elapsed: 0,2536841
09:00:08.00:  elapsed: 0,2091143
09:00:08.24:  elapsed: 0,2462239
09:00:08.50:  elapsed: 0,2605043
09:00:08.92:  elapsed: 0,4168645

.NET 8 app stat:

09:02:07.58:  elapsed: 1,9920193
09:02:07.98:  elapsed: 0,3940461
09:02:29.30: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
09:02:50.40: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
09:03:11.49: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
09:03:11.85:  elapsed: 0,3618588
09:03:12.25:  elapsed: 0,398727
09:03:33.49: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
09:03:54.61: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
09:04:15.71: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

[Bykiev]

@mconnew, we did some research and found that first request is made without Authorization header, then server sends second request with included Authorization header. It seems it breaks the server or some additional checks occured which cause requests block for some time (WAF, for example). With .NET Framework there is no first request without auth headers and that's why it's working fine. Is any ideas?

[Bykiev]

It seems it's a bug in WCF, the first request shouldn't be anonymous, because I've already specified the auth scheme via
binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;
and it should made only one request with basic auth headers

[Bykiev]

Hi, is any feedback on this?

[Bykiev]

@mconnew, can you please check this?

[mconnew]

This is a change in behavior between HttpWebRequest and HttpClientHandler/SocketsHttpHandler when setting PreAuthenticate to true. There's not much that WCF can do to change this. As the Basic auth mechanism is so simple (It's username:password that's base64 encoded), you could add the auth header yourself via a message inspector to work around this limitation of the .NET Http implementation.

[Bykiev]

As a workaround we are alresdy adding the header manually. If it can't be fixed in WCF, maybe some notice in the docs should be added?