OutOfMemoryException after sending a lot of huge messages

[m-jepson]

Hi,

I have a WPF application, which reads images from a webcam, does some detection and calculation on them and sends the results to a web application as an interface.
One of the results is the image itself, which is sent as a base64 string, using the following:

internal static void SendImage(string base64Image)
{
  IHubContext context = GlobalHost.ConnectionManager.GetHubContext<ScannerHub>();
  context.Clients.All.updateImage(base64Image);
}

The images are about 50kB in size, and are sent (because of testing) at 5 fps, to a client on the local network, so network speed is not a problem.
But ... for some reason the memory footprint of the application goes through the roof, reaching over 1GB of memory in use, shortly after which I get an OutOfMemoryException (even though my machine has 16GB of memory).
If I comment out the line context.Clients.All.updateImage(base64Image);, everything works fine and the memory footprint varies between 55 and 75 MB.

So I guess it's the message queue / store of SignalR which is filling the memory, but I found somewhere that only the last 30 seconds of messages are queued, so how can it still be growing after a couple of minutes?

My best solution would be to drop the message store altogether for this message type, as I'd rather see the client drop a few frames than showing old frames as it is trying to catch up when the connection speed is too slow.

I'm using VS 2012 on a Windows 8 development machine, using a self built version of SignalR 1.0.0alpha2, using the "Server" object from the Microsoft.Aspnet.SignalR.Self namespace to host the SignalR server in my WPF application.
I use a Hub object, with a method as specified above,which is the only method it has at the moment, but it will get some additional methods through which the client can control the WPF application.

Any help fixing the memory problem, purging the message queue and / or disabling the queue would be very much appreciated!

Regards,
Michael Jepson

[davidfowl]

Unfortunately you can't change the size of the message buffer right now. We have an open issue to make it configuration but I'm surprised you're running out of memory. Do you have a stack trace? How many clients are you broadcasting to?

[davidfowl]

Also, what transport are you using?

[m-jepson]

It runs out of memory even when not broadcasting to any connected client at all (unless I only send a new image when the client indicates it wants a new one of course).
I don't have a stack trace of the problem itself, as the problem occurs at random locations - wherever the first object is created that won't fit in memory.

As to what transport I am using: how do I check this? I use Google Chrome as client (if I connect any client) and default settings for SignalR itself.

[m-jepson]

Hmm, even weirder: I have added a checkbox in my WPF application, with which to toggle broadcasting images. If I leave it off, memory usage remains below 75 MB. If I switch it on, it starts going up slowly, but increasing in speed.
I have let it go to about 300 MB and then I switched off the broadcast. I could see my browser no longer receiving any images.
But, strangely, the memory footprint kept on growing! It kept on growing up to about 500 MB (in about 2 minutes), after which it suddenly dropped back to 65 MB.

[m-jepson]

Might hosting the hub through Owin make a difference? I cannot find a guide on how to do so though, but if you have some tips for this, I'm willing to give it a try!

[davidfowl]

I don't see how it would make a difference. We don't even know where the leak is so I can't propose a solution to a problem we haven't figured out.

[m-jepson]

I thought maybe the Server object was taking care of the caching / queueing. If that is the case, using another way of hosting the hub might make a difference. Just a thought.

[davidfowl]

Can you make a repro project that I can run?

[m-jepson]

I have created a small project, which can be found here: http://ge.tt/9zNUWKT

It searches for the dll's of SignalR in "D:\My Files\My Downloads\SignalR-1.0alpha2..." so you might want to change this. I have included a complete build, which will have the dll's as I have used them in the target folder.

The build is 32 bits, as that will be my target for the final product (due to other dependencies). I have tested it and it also shows the memory problem. Hopefully it will reproduce on your machine.

It shows an (empty) WPF form, but that is just there to better resemble the actual project that is having these problems.

[davidfowl]

So I just run it and wait for it to blow up?

[m-jepson]

Yes, sorry, forgot to mention. It wil just read the image from disk and send it out through the hub 5 times per second (approx.). Even with no client connected it will blow up, at least on my Win8 dev machine.

[davidfowl]

How long does it take?

[davidfowl]

Can you come to https://jabbr.net/#/rooms/signalr

[davidfowl]

So after much debugging. The rooted strings in the message buffer are what cause the memory to stick around. I'm adding a DefaultMessageBufferSize option to allow tweaking the buffer size globally but not per signal.

I verified this on your repro and it reduces memory allocations. You can estimate memory use to be :
Sum( signal (hub/ specific connection etc.) * DefaultMessageBufferSize )

At peak. I tried your repro with DefaultMessageBufferSize = 100 and it worked well. This change will go into a later release as the rc build is already locked down. Beware of buffer sizes that are too small as you might miss messages broadcast rate is high enough.

It will be inefficient to send large strings through signalr in general. For best performance/memory use, it's best to break your messages up into smaller chunks (32K max) and send those across the wire.

[davidfowl]

Fixed in 48dc366

[m-jepson]

You cannot change it yet. What you need to do is make sure that you don't send messages through SignalR that will result in a JSon serialised string of more than about 35k characters, as a string object of more than 85k bytes in memory will end up in the LOH (remember, each char is 16 bytes in .Net).

What I did: I used a StringBuilder to hold my complete Base64 string and used its "ToString(int, int)" function to get substrings to send to the client.
At the client side, I concatenate these parts.
As a piece of advise: Send the amount of parts along and create an Array of the right size, also send a parts index along, so you can put in the the correct location in the Array (e.g. parts[index] = receivedString). This will prevent problems if for some reason two parts arrive in reverse order.
After that, use the Array's join() method to get the whole Base64 string on the client (e.g. parts.Joint("")).

And an extra piece of advise: In order to prevent slower connections from getting a backlog on images, have the client send a notification back to the server on complete reception of an image and have your server only send a new image to clients that have completed their last image. Use Clients.AllExcept to send only to those clients that are not still busy with receiving an earlier image.

[davidfowl]

@m-jepson you should write a blog post on that 😄. BTW in the next release you'll be able to tweak the buffer size.

[rustd]

verified