Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

SB scale-out with multiple streams Send failed "Object reference not set to an instance of an object" #2933

Closed
Xiaohongt opened this Issue · 2 comments

2 participants

@Xiaohongt
Collaborator

Run Stress tests with /Senders:2 /Run:ServiceBusMessageBus /ServiceBusConnectionString:, we can see below exceptions in trace:

SignalR.ScaleoutMessageBus Error: 0 : Stream(0) - Send failed: System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNet.SignalR.ServiceBus.ServiceBusMessageBus.Send(Int32 streamIndex, IList`1 messages) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.ServiceBus\ServiceBusMessageBus.cs:line 64
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.Send(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 81
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.<Send>b__0(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 66
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStream.SendContext.InvokeSend() in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStream.cs:line 299
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNet.SignalR.ServiceBus.ServiceBusMessageBus.Send(Int32 streamIndex, IList`1 messages) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.ServiceBus\ServiceBusMessageBus.cs:line 64
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.Send(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 81
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.<Send>b__0(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 66
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStream.SendContext.InvokeSend() in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStream.cs:line 299

Looks like it is caused by this:
When use multiple streams, e.g. when Stream(0) become open, and in Scale-out Subscribe is not done, in the SB scale-out case, _connectionContext is null for Subscribe, but in Core, since Stream(0) is open, for Stream(0), ScaleoutStream.Send will call .ScaleoutStream.SendContext.InvokeSend() which will call _connectionContext.Publish(streamIndex, stream) in ServiceBusMessageBus.cs, then we see the exception

I tried below fix to use new Task to run Retry() in class ServiceBusConnection , it works and this issue not repro anymore.
Not sure if Task.Yield is needed or not in Retry() method?

        public ServiceBusConnectionContext Subscribe(IList<string> topicNames,
                                                     Action<int, IEnumerable<BrokeredMessage>> handler,
                                                     Action<int, Exception> errorHandler,
                                                     Action<int> openStream)
        {
            . . . . . . .

            for (var topicIndex = 0; topicIndex < topicNames.Count; ++topicIndex)
            {
                var topic = topicIndex;
               Task.Run(() => Retry(() => CreateTopic(connectionContext, topic)));
            }
            . . . . . . .
        }

        private async Task Retry(Action action)
       {
            await Task.Yield();

            string errorMessage = "Failed to create service bus subscription or topic : {0}";
            while (true)
            {
            . . . . . . 
        }

Probably in class ServiceBusMessageBus, add try-catch in Dispose(bool disposing):

        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);

            if (disposing)
            {
                if (_connectionContext != null)
                {
                    try
                    {
                        _connectionContext.Dispose();
                    }
                    catch { };
                }
               . . . . . . 
            }
        }

Note:
With #2767 fix, then we can see this issue repro very easily, we should also fix this in 2.0.3

@Xiaohongt Xiaohongt added the bug label
@DamianEdwards DamianEdwards referenced this issue from a commit
@DamianEdwards DamianEdwards Fix for null ref exception in Service Bus scale-out:
- The ConnectionContext is now created immediately and passed in to the Subscribe call for initialization, so it should never be null
- #2933
0e84a36
@DamianEdwards DamianEdwards added this to the 2.0.3 milestone
@DamianEdwards

@Xiaohongt this is fixed now. Pls verify and close out.

@Xiaohongt
Collaborator

Verified that SB scale-out with multiple streams System.NullReferenceException don't repro anymore from Send.

@Xiaohongt Xiaohongt closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.