Infinite loop in BatchingMessageSender when SendBatch(toSend) fails

[michaelnoonan]

We've experienced several times where a server will go into an endless loop until it crashes with an OutOfMemoryException (about once per day over the last week). I've finally been able to reproduce the issue and I have a theory about the cause.

We've got 600MB of the following log message:

2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)
2014-05-08 07:36:59.537 +00:00 [Debug] Flushing outbound message queue "q.mymessage" (1 messages)

Looking at the code in https://github.com/NimbusAPI/Nimbus/blob/master/src/Nimbus/Infrastructure/MessageSendersAndReceivers/BatchingMessageSender.cs#L75

If the underlying MessageSender throws immediately, say it gets disposed, the BatchingMessageSender will go into an endless loop and never recover because the MessageSender isn't recycled.

A couple of thoughts:

1. It sounds like this is part of a larger issue on recycling messaging clients
2. Is the RetryPolicy.Default (exponential backoff) in the MessageSender not sufficient?