ASP.NET Core SignalR messages are stalled for hours and lost

[keremkambur]

Bug Description

We are using SignalR to communicate between servers. We have an individual API that sends messages using group connection and if the destination API couldn't update a reference table that it received the message based on an identifier we pass, we establish a direct connection to that specific destination.

However, in some instances API that sends the messages doesn't respond for hours. Here below you may see an example request and its ultimate end.

2021-08-24 17:00:07.877 +00:00 [Information] Request starting HTTP/1.1 GET http://{serverBaseUri}/cacheUpdateHub?GroupName=Publisher&InstanceId={azureResourceName}-82cecef0&ApplicationName={azureResourceName}&MachineName=SandboxHost-637648365386280212&id=K34JTC9kfCLWJP_Gu-8gyQ

2021-08-24 18:50:44.065 +00:00 [Information] Request finished in 6636188.3742ms 101

We simply followed https://docs.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/working-with-groups and https://docs.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/mapping-users-to-connections#groups guides during the implementation. We are using our Azure resource names/instance IDs/machine names as the group and direct connection identifiers. Those messages are actually lost and the connection id K34JTC9kfCLWJP_Gu-8gyQ doesn't exist anywhere else except above in the logs.

To have more logs, we have added the below providers but they added nothing to the logs.

"Microsoft.AspNetCore.SignalR": "Debug",
"Microsoft.AspNetCore.Http.Connections": "Debug"

Further technical details

"Microsoft.AspNetCore.SignalR.Client" Version="3.1.7" "Microsoft.AspNetCore.SignalR" Version="1.1.0"`

Sender API: netcoreapp3.1
All listener APIs: netcoreapp3.1

Logs

If it's needed, some successful messages can also be shared as well as some end-to-end transaction logs from Application Insights. However, I didn't find them informing so didn't want to put them before it's asked.

[davidfowl]

When you say messages are stalled for hours, what exactly do you mean? Do you have any incoming calls into your hub, or are you using the hub for purely outgoing messages?

[keremkambur]

Thank you for the response. We are using the hub for only outgoing messages. As you may see in the log, the request takes hours to be finished (6636188.3742ms) and the message did not arrive.

Maybe the below piece of log also might be useful. We also log the connection IDs on OnConnected and OnDisconnected events. However, for the above example, the connection ID is never logged on either connection or disconnection events.

[[OnConnectedAsync] is called. ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ];]; Start
[OnConnectedAsync]; Adding group: [Publisher]
[OnConnectedAsync]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Live]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Live]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Live]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Live]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Test]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Test]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Test]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Test]
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Cms];
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Cms];
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Cms];
[NotifyGroup]; End
[[NotifyGroup] is called! ConnectionId: [ofJSby-qoy1UfhTGBMQmlQ]; Group: [Cms];
[NotifyGroup]; End

[davidfowl]

As you may see in the log, the request takes hours to be finished (6636188.3742ms) and the message did not arrive.

I don't know where that number is coming from, did you upload a log file somewhere in the issue? When you say the request takes hours to finish what do you mean? SignalR requests are long running, are you referring to something else?

I assume when you say the message didn't arrive, you mean on the client side right? Do you have client side logs captured that show the message not being delivered even though the server is sending it?

Can you show more of your code? What is notify group? How does it get called? What does the client code look like?

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.