LP/SSE hang when using a reasonable value for ServicePoint DefaultConnectionLimit #1874

Closed
jcondex opened this Issue Apr 15, 2013 · 1 comment

3 participants

@jcondex

Ever since we switched our stress tests to use reasonable (calculated) values for the ServicePoint defaultConnectionLimit queue, we started seeing systematic hangs on the client side which block most of our stress tests.

Previously we set ServicePointManager.DefaultConnectionLimit to int.MaxValue; which caused the tests to run for longer.

The problem seems to stem from a race condition in which a Poll is being queued after a Send.

Here is a basic repro for this issue:

Client:

using System;
using System.Net;
using System.Threading;
using Microsoft.AspNet.SignalR.Client.Hubs;
using Microsoft.AspNet.SignalR.Client.Transports;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var mre = new ManualResetEvent(false))
            {
                ServicePointManager.DefaultConnectionLimit = 2;
                var iteration = 0;
                var client = new HubConnection("http://CSVM2226712-TA/SignalR.Stress.SimpleEcho");
                var proxy = client.CreateHubProxy("SimpleEchoHub");

                proxy.On("AsyncEcho", id => 
                {
                    Console.WriteLine("Received id: {0}", id);
                    mre.Set();
                });

                client.Start(new LongPollingTransport()).Wait();

                while (true)
                {
                    Console.WriteLine("Invoking AsyncEcho");
                    proxy.Invoke("AsyncEcho", iteration++).Wait();
                    Console.WriteLine("Invoked AsyncEcho");
                    mre.WaitOne();
                    mre.Reset();
                }
            }
        }
    }
}

Hub:

using Microsoft.AspNet.SignalR;

namespace SignalR.Stress.SimpleEchoHub
{
    public class SimpleEchoHub : Hub
    {
        public void AsyncEcho(string str)
        {
            Clients.Caller.AsyncEcho(str);
        }
    }
}

Previously we had been tracking this issue by means of #1138 but given that WebSockets should not be affected by this I have decided to track it here.

@halter73 halter73 added a commit that referenced this issue Apr 17, 2013
@halter73 halter73 Don't use Task.Factory.FromAsync in StreamExtensions
- We have seen WriteAsync hang

#1874
458f6c2
@halter73 halter73 added a commit that referenced this issue Apr 17, 2013
@halter73 halter73 Don't use Task.Factory.FromAsync in StreamExtensions
- We have seen WriteAsync hang

#1874
0e0a571
@halter73 halter73 added a commit that referenced this issue Apr 17, 2013
@halter73 halter73 Don't use Task.Factory.FromAsync in StreamExtensions
- We have seen WriteAsync hang

#1874
d33dce3
@halter73 halter73 added a commit that referenced this issue Apr 17, 2013
@halter73 halter73 Don't use Task.Factory.FromAsync in StreamExtensions
- We have seen WriteAsync hang

#1874
f686354
@halter73 halter73 added a commit that referenced this issue Apr 17, 2013
@halter73 halter73 Fix hang in StreamExtensions.WriteAsync
- Don't use Task.Factory.FromAsync in StreamExtensions
- Use WriteAsync and ReadAsync directly in StreamExtensions on .NET 4.5

#1874
c85d41e
@jcondex jcondex closed this Apr 17, 2013
@davidfowl
SignalR member

So this bug wasn't fixed. The cause of the issue was a server bug deep within the message bus. I was able to reproduce this locally after @halter73 added tons of tracing to narrow down where the issue was. The fix is pretty trivial so I'll commit it.

@davidfowl davidfowl added a commit that referenced this issue Apr 23, 2013
@davidfowl davidfowl Fixed stress issue with hanging tests.
- Only add subscriptions to the topic after it's been fully setup.
- Added new stress run to send in a loop.

#1874
c2479a9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment