Connection errors that happens intermittently

[johandynamics]

We are using the nuget package in a service that calls Dynamics a few times a minute.

And intermittently we get errors that we couldn't connect because a few different exceptions:

1. Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse\r\n ---> System.AggregateException: One or more errors occurred. (Exception - Fault While initializing client - RefreshInstanceDetails)
2. Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse\r\n ---> System.AggregateException: One or more errors occurred. (Need a non-empty authority (Parameter 'Authority'))
3. Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse\r\n ---> System.AggregateException: One or more errors occurred. (An error occurred while sending the request.)\r\n ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.\r\n ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..\r\n ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

We also get
System.ServiceModel.CommunicationException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (domain.crm4.dynamics.com:443)
But not sure if that is connected to your nuget package.

This happens 2 or 3 times each day on different times, and then there can be a surge of up to 10 to 15 exceptions.

I'm sure there is a lot of information I have not included that you need, but just let me know what you need.

[wasiuddin7]

we are also facing the same problem in our solution when we use our application after 10 to 15 API calls getting the issue.
we posted the issue in the same platform issue 321, if you find any solution let us know.

[MattB-msft]

We have seen a few reports of this in the last few weeks.

The errors are indicating a network issue between the client and the server,
@johandynamics, on your #2, are there should be sub-exceptions there, most likely what you're seeing in #3 Can you validate?

What may be needed here is to create a Support ticket to engage Platform networking folks to help troubleshoot this problem specifically to the environments your connecting too.

[johandynamics]

This how the errors look when I grab more of them. At #2 it looks like the first inner stack trace is similar to #3 second inner stack trace. But I don't really know what I'm looking for so I hope this is the information you wanted @MattB-msft

#2
"Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse\r\n
---> System.AggregateException: One or more errors occurred. (Need a non-empty authority (Parameter 'Authority'))\r\n
---> System.ArgumentNullException: Need a non-empty authority (Parameter 'Authority')\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.InitServiceAsync()\r\n
--- End of inner exception stack trace ---\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)\r\n
--- End of inner exception stack trace ---\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.ConnectToService(String connectionString, ILogger logger)\r\n

#3
"Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse\r\n
---> System.AggregateException: One or more errors occurred. (An error occurred while sending the request.)\r\n
---> System.Net.Http.HttpRequestException: An error occurred while sending the request.\r\n
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..\r\n
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.\r\n
--- End of inner exception stack trace ---\r\n
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)\r\n
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)\r\n
at System.Net.Http.HttpConnection.g__ReadAheadWithZeroByteReadAsync|44_0()\r\n
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n
--- End of inner exception stack trace ---\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.InitServiceAsync()\r\n
--- End of inner exception stack trace ---\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)\r\n
--- End of inner exception stack trace ---\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)\r\n
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.ConnectToService(String connectionString, ILogger logger)\r\n

[MattB-msft]

Yes.. this is what I was expecting to see.

This is your error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

That is the network layer terminating the connection, that can happen due to bad gateway's or firewalls, connection drops (mid-communication)

We have discussed putting in retires for this type of error, but our telemetry tells us that when we see "An existing connection was forcibly closed by the remote host" it nearly aways a network element (firewall / gateway / proxy server ) that is failing. Retrying the operation internally masks the problem and shows up as performance issues.

I would start by taking a look at where your service is and the path it takes to get to PowerPlatform. You can also aways open a support ticket with us to help troubleshoot the problem from the PowerPlatform side to your host (where possible).

That said, this behavior has occurred in the past where our (PowerPlatform) perimeter security system would terminate connections due to out of date or what were seen as corrupted SSL certificates used on requests. But I, personally, have not seen that since we retired TLS 1.x in the network.

My recommendation would be as follows:
If you are hosting this client locally (your own vm / local host) make sure it's running a supported OS and that OS is patched current (Linux / windows / mac / whatever), if your still seeing the issue after that reach out to the team that owns the firewall/gateway/proxy server managing traffic and see if they can help you troubleshoot it there, connection terminations should appear in their logs. I have seen this, in the past, reveal a bad node or a proxy server that was misconfigured, which would be supported in this case by the semi randomness its being seen.

If that does not pan out, or lead to a reason, I would recommend opening a support ticket with Microsoft to so the PowerPlatform side can be investigated.

[johandynamics]

@MattB-msft thank you, we'll start with your recommendations and if the issue still persists we'll open a support ticket with Microsoft and I'm guessing in our case it's the Dynamics365 team for us.

[MattB-msft]

Going to close this..
thanks !

[mohsinonxrm]

BTW, just want to mention that we're seeing error 2 and 3 from the OP.

We were using .NET framework 4.6.2 along with Xrm Tooling and no errors, but when we deployed .NET 6 using DV Client (latest) we've started seeing these exceptions as well.

Aside from this, I have Azure Functions (.NET 6, v4 runtime) and getting the SSL exception message intermittently. They were running just fine for the past few months but recently I've been seeing these errors now.

IMO, I don't believe it is a network configuration, otherwise we would've seen the same errors with .NET framework 4.6.2 with the XRM tooling connector as well.

I can't believe we're the only ones reporting or having these.

@MattB-msft , happy to provide additional information, please let me know what you need?

[rsaclee]

We are experience similar issue. We used SDK to connect to CRM on cloud. Per Microsoft recommendation, we replace ADAL auth with Microsoft.PowerPlatform.Dataverse.Client. .NET application is running on framework 4.7.2. We did not get any error when doing unit testing in our test environment. However, after we deployed the application to our production with higher volume of transactions (we load more than 1/2 million of records daily), then we got the following error

Exception Message:
Failed to connect to Dataverse
StackTrace:
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient..ctor(Uri instanceUrl, String clientId, String clientSecret, Boolean useUniqueInstance, ILogger logger)

Inner Message:
One or more errors occurred.
Inner StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result()
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject)
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject)
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.DoLogin(ConnectionService& ConnectionObject)
at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath)

After rolling back the changes to use Microsoft.Xrm.Tooling.Connector with ADAL, the error is gone.

Any help or comment for trouble shooting is appreciated.

[mohsinonxrm]

BTW, just an update here. In me case, it actually did turn out to be a customer's fairly complex network configuration being the issue. Anyway, we were able to replicate and compare side by side and they know what to do or how to fix.

@rsaclee , I'm not sure the latest issue you're having is related to the OP.

@MattB-msft , thanks as always. It was indeed a network issue on the customer side.