Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Performance problem using long polling with many clients #2505

Closed
halter73 opened this Issue · 2 comments

3 participants

@halter73
Collaborator

Using the HubConnectionAPI sample, slightly modified as follows. This is hosted in an Azure cloud service.

  • In the javascript, when a disconnect occurs it will kick off a randomized timer to reconnect the client
  • It broadcasts a message to all clients every 10 seconds
  • Removed the broadcasts in OnConnected, OnReconnected, OnDisconnected.

Using PhantomJs, I created 900 clients that navigate to this page adding '?transport=longPolling' to the end of the url to force long polling. (150 instances per VM, 6 large VMs in Azure)

All 900 clients connect cleanly and the processor load on the server is fairly low.

I then stop IIS on the server, wait for 3 minutes, then start IIS. The processor load then shoots up to 100% and pretty well stays there. See the attached image. The spikes on the left correspond to the broadcasts every 10 seconds, growing larger as more clients connect. The low processor usage in the middle is when IIS is stopped. The 100% processor usage area on the right is when I restart IIS. It doesn't matter how long I leave it, the processor time usage will never reduce, but the clients all still seem to work.

cpuprofile
I attached a debugger while IIS was stopped, so have a log file which shows the 900 clients reconnecting which is available here http://sdrv.ms/Zyp1TI There are also two dotTrace files available there, but they didn't really give me any clues.

The changes to HubConnectionAPI.js are as follows. Notice that the call to $.connection.hub.start is factored out into a function which is called from a timer in the handler for $.connection.hub.disconnected

/// <reference path="../../Scripts/jquery-1.8.2.js" />
$(function () {
    "use strict";

    var hubConnectionAPI = $.connection.hubConnectionAPI,
        messages = $("#messages"),
        groupNameTextInput = $("#group"),
        connectionTextInput = $("#connection"),
        messageTextInput = $("#message"),
        groupMessageTextInput = $("#groupMessage"),
        meTextInput = $("#me"),
        stopStartBtn = $("#stopStart"),
        start;

    $.connection.hub.logging = true;

    hubConnectionAPI.client.displayMessage = function (value) {
        $("<li/>").html("[" + new Date().toTimeString() + "]: " + value).appendTo(messages);
    }

    $.connection.hub.stateChanged(function (change) {
        var oldState = null,
            newState = null;

        for (var state in $.signalR.connectionState) {
            if ($.signalR.connectionState[state] === change.oldState) {
                oldState = state;
            } else if ($.signalR.connectionState[state] === change.newState) {
                newState = state;
            }
        }

        $("<li/>").html("[" + new Date().toTimeString() + "]: " + oldState + " => " + newState + " " + $.connection.hub.id)
                    .appendTo(messages);
    });

    $.connection.hub.reconnected(function () {
        $("<li/>").css("background-color", "green")
                    .css("color", "white")
                    .html("[" + new Date().toTimeString() + "]: Connection re-established")
                    .appendTo(messages);
        $("<li/>").html("reconnected transport: " + $.connection.hub.transport.name + " " + $.connection.hub.id)
                    .appendTo(messages);
    });

    $.connection.hub.error(function (err) {
        $("<li/>").html("Error occurred: " + err).appendTo(messages);
    });

    $.connection.hub.connectionSlow(function () {
        $("<li/>").html("[" + new Date().toTimeString() + "]: Connection Slow").appendTo(messages);
    });

    var reconnectTimeoutHandle = null;
    $.connection.hub.disconnected(function () {
        stopStartBtn.prop("disabled", false)
                    .find("span")
                        .text("Start Connection")
                        .end()
                    .find("i")
                        .removeClass("icon-stop")
                        .addClass("icon-play");
        window.clearTimeout(reconnectTimeoutHandle);
        reconnectTimeoutHandle = window.setTimeout(start, (Math.random() * 10000) + 2000);
    });

    start = function () {
        $.connection.hub.start({ transport: activeTransport, jsonp: isJsonp })
            .done(function () {
                $("<li/>").html("started transport: " + $.connection.hub.transport.name + " " + $.connection.hub.id)
                            .appendTo(messages);

                stopStartBtn.prop("disabled", false)
                            .find("span")
                                .text("Stop Connection")
                                .end()
                            .find("i")
                                .removeClass("icon-play")
                                .addClass("icon-stop");
            });
    };
    start();


    $('#joinGroup').click(function () {
        // Set the connection Id to the specified value or the generated SignalR value
        var connectionIdToJoin = connectionTextInput.val() || $.connection.hub.id;

        hubConnectionAPI.server.joinGroup(connectionIdToJoin, groupNameTextInput.val()).done(function (value1) {
            $("<li/>").html("Succeeded at joinGroup: " + value1).appendTo(messages);
        }).fail(function (e) {
            $("<li/>").html("Failed at joinGroup: " + e).appendTo(messages);
        });
    });

    $('#leaveGroup').click(function () {
        // Set the connection Id to the specified value or the generated SignalR value
        var connectionIdToLeave = connectionTextInput.val() || $.connection.hub.id;

        hubConnectionAPI.server.leaveGroup(connectionIdToLeave, groupNameTextInput.val()).done(function (value1) {
            $("<li/>").html("Succeeded at leaveGroup: " + value1).appendTo(messages);
        }).fail(function (e) {
            $("<li/>").html("Failed at leaveGroup: " + e).appendTo(messages);
        });
    });

    $("#broadcast").click(function () {
        hubConnectionAPI.server.displayMessageAll(messageTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at getMessage: " + e).appendTo(messages);
        });
    });

    $("#broadcastExceptSpecified").click(function () {
        hubConnectionAPI.server.displayMessageAllExcept(messageTextInput.val(), connectionTextInput.val().split(",")).fail(function (e) {
            $("<li/>").html("Failed at getMessageAllExcept: " + e).appendTo(messages);
        });
    });


    $("#other").click(function () {
        hubConnectionAPI.server.displayMessageOther(messageTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at getMessageOther: " + e).appendTo(messages);
        });
    });


    $("#sendToMe").click(function () {
        hubConnectionAPI.server.displayMessageCaller(meTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at getMessageCaller: " + e).appendTo(messages);
        });
    });


    $("#specified").click(function () {
        hubConnectionAPI.server.displayMessageSpecified(connectionTextInput.val(), meTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at getMessageSpecified: " + e).appendTo(messages);
        });
    });

    $("#groupmsg").click(function () {
        hubConnectionAPI.server.displayMessageGroup(groupNameTextInput.val(), groupMessageTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at getMessageGroup: " + e).appendTo(messages);
        });
    });

    $("#groupmsgExceptSpecified").click(function () {
        hubConnectionAPI.server.displayMessageGroupExcept(groupNameTextInput.val(), groupMessageTextInput.val(), connectionTextInput.val().split(",")).fail(function (e) {
            $("<li/>").html("Failed at displayMessageGroupExcept: " + e).appendTo(messages);
        });
    });

    $("#otherInGroupmsg").click(function () {
        hubConnectionAPI.server.displayMessageOthersInGroup(groupNameTextInput.val(), groupMessageTextInput.val()).fail(function (e) {
            $("<li/>").html("Failed at displayMessageOthersInGroup: " + e).appendTo(messages);
        });
    });

    stopStartBtn.click(function () {
        var $el = $(this);

        $el.prop("disabled", true);

        if ($.trim($el.find("span").text()) === "Stop Connection") {
            $.connection.hub.stop();
        } else {
            start();
        }
    });
});

The changes to HubConnectionAPI.cs are as follows. Notice the removal of the OnConnected, OnDisconnected and OnReconnected overrides, and the addition of a few static bits to send a message every 10 seconds.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Timers;
using System.Linq;
using System.Collections.Concurrent;
using System.IO;

namespace Microsoft.AspNet.SignalR.Hosting.AspNet.Samples.Hubs.HubConnectionAPI
{
    public class PingData
    {
        public string connId;
        public DateTime date;
    }

    public class HubConnectionAPI : Hub
    {
        static IHubContext context;
        static Timer t;
        static HubConnectionAPI()
        {
            t = new Timer(10000);
            t.Elapsed += SendMessage;
            t.Start();
            context = GlobalHost.ConnectionManager.GetHubContext<HubConnectionAPI>();
        }

        static void SendMessage(object sender, ElapsedEventArgs e)
        {
            context.Clients.All.displayMessage("Hello");
        }

        public string JoinGroup(string connectionId, string groupName)
        {
            Groups.Add(connectionId, groupName).Wait();
            return connectionId + " joined " + groupName;
        }

        public string LeaveGroup(string connectionId, string groupName)
        {
            Groups.Remove(connectionId, groupName).Wait();
            return connectionId + " removed " + groupName;
        }

        public void DisplayMessageAll(string message)
        {
            Clients.All.displayMessage("Clients.All: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageAllExcept(string message, params string[] excludeConnectionIds)
        {
            Clients.AllExcept(excludeConnectionIds).displayMessage("Clients.AllExcept: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageOther(string message)
        {
            Clients.Others.displayMessage("Clients.Others: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageCaller(string message)
        {
            Clients.Caller.displayMessage("Clients.Caller: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageSpecified(string targetConnectionId, string message)
        {
            Clients.Client(targetConnectionId).displayMessage("Clients.Client: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageGroup(string groupName, string message)
        {
            Clients.Group(groupName).displayMessage("Clients.Group: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageGroupExcept(string groupName, string message, params string[] excludeConnectionIds)
        {
            Clients.Group(groupName, excludeConnectionIds).displayMessage("Clients.Group: " + message + " from " + Context.ConnectionId);
        }

        public void DisplayMessageOthersInGroup(string groupName, string message)
        {
            Clients.OthersInGroup(groupName).displayMessage("Clients.OthersInGroup: " + message + " from" + Context.ConnectionId);
        }
    }
}

#2125

@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Captured several long polling timeouts that are then cleared on stop.
- Also added two fields to the private data field of a connection object.

#2505
acb382a
@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Addressed code review comments.
- Added an extra check after long polling error handler trigger to ensure that multiple connections do not occur.

#2505
8c31f57
@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Captured several long polling timeouts that are then cleared on stop.
- Also added two fields to the private data field of a connection object.

#2505
14218ee
@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Addressed code review comments.
- Added an extra check after long polling error handler trigger to ensure that multiple connections do not occur.

#2505
73a4f7b
@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Captured several long polling timeouts that are then cleared on stop.
- Also added two fields to the private data field of a connection object.

#2505
04ded87
@NTaylorMullen NTaylorMullen referenced this issue from a commit
@NTaylorMullen NTaylorMullen Addressed code review comments.
- Added an extra check after long polling error handler trigger to ensure that multiple connections do not occur.

#2505
16379f4
@gustavo-armenta

tested 5 longPolling connections, stopped server, waited for clients to trigger reconnect, started server, clients reconnected, broadcasted one message. CPU goes down to 10% and network is not sending traffic just waiting on "signalr/poll" requests to complete.

@NTaylorMullen: release1.1.4 does not have connectionData on "/signalr/negotiate". Is it expected? I see its fixed for release1.1.5

@gustavo-armenta

Taylor response:
That's correct, we're not porting that fix to 1.1.4, we are porting it to 1.1.5 though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.