SB scale-out with multiple streams Send failed "Object reference not set to an instance of an object" #2933

Closed
Xiaohongt opened this Issue Mar 14, 2014 · 2 comments

Comments

Projects
None yet
2 participants
@Xiaohongt
Contributor

Xiaohongt commented Mar 14, 2014

Run Stress tests with /Senders:2 /Run:ServiceBusMessageBus /ServiceBusConnectionString:, we can see below exceptions in trace:

SignalR.ScaleoutMessageBus Error: 0 : Stream(0) - Send failed: System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNet.SignalR.ServiceBus.ServiceBusMessageBus.Send(Int32 streamIndex, IList`1 messages) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.ServiceBus\ServiceBusMessageBus.cs:line 64
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.Send(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 81
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.<Send>b__0(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 66
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStream.SendContext.InvokeSend() in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStream.cs:line 299
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNet.SignalR.ServiceBus.ServiceBusMessageBus.Send(Int32 streamIndex, IList`1 messages) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.ServiceBus\ServiceBusMessageBus.cs:line 64
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.Send(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 81
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStreamManager.<Send>b__0(Object state) in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStreamManager.cs:line 66
   at Microsoft.AspNet.SignalR.Messaging.ScaleoutStream.SendContext.InvokeSend() in d:\dd\SignalR_Release_2.0\SignalR\src\Microsoft.AspNet.SignalR.Core\Messaging\ScaleoutStream.cs:line 299

Looks like it is caused by this:
When use multiple streams, e.g. when Stream(0) become open, and in Scale-out Subscribe is not done, in the SB scale-out case, _connectionContext is null for Subscribe, but in Core, since Stream(0) is open, for Stream(0), ScaleoutStream.Send will call .ScaleoutStream.SendContext.InvokeSend() which will call _connectionContext.Publish(streamIndex, stream) in ServiceBusMessageBus.cs, then we see the exception

I tried below fix to use new Task to run Retry() in class ServiceBusConnection , it works and this issue not repro anymore.
Not sure if Task.Yield is needed or not in Retry() method?

        public ServiceBusConnectionContext Subscribe(IList<string> topicNames,
                                                     Action<int, IEnumerable<BrokeredMessage>> handler,
                                                     Action<int, Exception> errorHandler,
                                                     Action<int> openStream)
        {
            . . . . . . .

            for (var topicIndex = 0; topicIndex < topicNames.Count; ++topicIndex)
            {
                var topic = topicIndex;
               Task.Run(() => Retry(() => CreateTopic(connectionContext, topic)));
            }
            . . . . . . .
        }

        private async Task Retry(Action action)
       {
            await Task.Yield();

            string errorMessage = "Failed to create service bus subscription or topic : {0}";
            while (true)
            {
            . . . . . . 
        }

Probably in class ServiceBusMessageBus, add try-catch in Dispose(bool disposing):

        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);

            if (disposing)
            {
                if (_connectionContext != null)
                {
                    try
                    {
                        _connectionContext.Dispose();
                    }
                    catch { };
                }
               . . . . . . 
            }
        }

Note:
With #2767 fix, then we can see this issue repro very easily, we should also fix this in 2.0.3

@Xiaohongt Xiaohongt added the bug label Mar 14, 2014

DamianEdwards added a commit that referenced this issue Mar 17, 2014

Fix for null ref exception in Service Bus scale-out:
- The ConnectionContext is now created immediately and passed in to the Subscribe call for initialization, so it should never be null
- #2933

@DamianEdwards DamianEdwards added this to the 2.0.3 milestone Mar 17, 2014

@DamianEdwards

This comment has been minimized.

Show comment Hide comment
@DamianEdwards

DamianEdwards Mar 17, 2014

Member

@Xiaohongt this is fixed now. Pls verify and close out.

Member

DamianEdwards commented Mar 17, 2014

@Xiaohongt this is fixed now. Pls verify and close out.

@Xiaohongt

This comment has been minimized.

Show comment Hide comment
@Xiaohongt

Xiaohongt Mar 18, 2014

Contributor

Verified that SB scale-out with multiple streams System.NullReferenceException don't repro anymore from Send.

Contributor

Xiaohongt commented Mar 18, 2014

Verified that SB scale-out with multiple streams System.NullReferenceException don't repro anymore from Send.

@Xiaohongt Xiaohongt closed this Mar 18, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment