SignalR: The server disconnected before sending a handshake response

[mhardy]

I have a Blazor Server app that has been working perfect with a small group of 10 users. I'm hosting it on a B1 App Service Plan and using an Azure SignalR Standard Service with 1 Unit. I just rolled it out 200 people, and it only took a few dozen to cause hundreds of exceptions.
Memory never exceeds 60%, SignalR service has 120 max client connections, 16 max server connections.

I have SignalR debug logging enabled. Digging through the exceptions I'm seeing hundreds of these:

The server disconnected before sending a handshake response

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

I can't reproduce the issue locally because it only occurs under load.

.NET Core 3.1
"Microsoft.AspNetCore.SignalR.Client" Version="3.1.8"
"Microsoft.Azure.SignalR" Version="1.5.1"

[BrennanConroy]

cc @vicancy Are there logs on the service that customers can see?

Do you have a repro app that we can use?

[mhardy]

No logs the customer can see. I can't repro it until I have a traffic spike and it starts happening. I can give you a demo login to the site, but it may or may not produce the error. It just happened again but I restarted the App and SignalR services and it's working again - for the time being. I expect a spike at 3:30 where I fear it will start happening again.

[vicancy]

what is the CPU usage of the app server side?

[mhardy]

Average is very low, no more than 11%. But spikes briefly to 80-90%. These do correlate to the times the exceptions start flooding the logs. I just scaled up my app service. I'll keep a closer eye on CPU usage. I was focused more on the SignalR service. I pushed it all the way up to 20 Units during the spike but it had no effect.

[mhardy]

Actually, that was average. If I change the aggregate to max it looks like this...

However the exceptions started at 2:01 PM. 16 minutes before the first 100% spike.

[vicancy]

The server disconnected before sending a handshake response

the server might be busy handling other things and failed the 15s ping-pong with service and as a result disconnected with the Azure SignalR Service.

Let's keep an eye on the status to see if scaling up the app server helps.

[mhardy]

I've been attempting some load testing. I was able to reproduce the exceptions by throwing artillery at it. Scaling up the App Service fixed the issue. I was able to repro the failures several times, and repro the scaled up fix also several times.

The problem I now face is determining what actual load the system can support. Although artillery was able to repro the failures, I'm not sure it's a valid test to determine scale since it doesn't keep connections open, so the SignalR message count is virtually 0 during tests.

I looked into Crank but "Performance counters are not available on Azure App Service Web Apps". Although that article is 5 years old so I'm not sure of it's relevance.

As a comparison, I created two razor pages identical to the razor components being tested but without SignalR (same exact database calls, etc.). These two test results were repro'd many times.

Elapsed	duration	arrivalRate	Errors	p95	p99	rps	requests	SignalR
23:11	1200	2	369	97,271	97,271	5.18	7,200	yes
20:05	1200	2	0	553	553	5.98	7,200	no

(duration and arrivalRate are the artillery phase settings, p95 & p99 are the latency values, rps=requests/sec)

As you can see the latency by introducing SignalR, even using the dedicated SignalR Service, is significant. Which I assume is what causes the "connection attempt" exceptions.

Here's a screenshot of app insights during the failures, showing the CPU higher, but not pinned.

CPU & Memory directly from the App Service Plan metrics during the test:

So the outstanding question is, what is the best way to load test a Blazor Server app that uses the SignalR Service? I need this data before I can re-launch to a wider audience.