OnMessage Handler

[jtaubensee]

@vinaysurya to provide more details

[SeanFeldman]

👍

[vinaysurya]

API Surface

public void OnMessageAsync(Func<BrokeredMessage, Task> callback);
public void OnMessageAsync(Func<BrokeredMessage, Task> callback, OnMessageOptions onMessageOptions);

OnMessageOptions link:
https://docs.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.onmessageoptions

Defaults:
MaxConcurrentCalls is 1
AutoRenewTimeout is set to 5 mins
AutoComplete is set to true

Other User facing Contracts:
OnMessageOptions.ExceptionReceived Event handler should be called when an operation in the Pump fails (say while receiving message/renewing lock on message/completing the message if AutoComplete is enabled/Abandon fails/ User Callback fails)

The ExceptionReceived is a FYI only contract that is primarily used for debugging purposes to identify what kind of failures are hapenning on the Pump. The Pump should handle all the renew/abandon/retry operations internally during failures.

Description:
The OnMessageAsync handler will take a User supplied callback and create a new MessageReceiverPump on its invocation. The Pump, as long as the receiver(associated with the QueueClient) is Open will keep trying to receive the message and for each message received it will in a seperate thread call for the Dispatch Task of that message. The Dispatch will inturn Schedule a RenewLock task for the message (on a different thread with a cancellation token) as specified by the OnMessageOptions and then call the User specified callback. Once the User callback is completed the Dispatch will cancel the renewloop task using the cancellation token. If there is an exception with executing User Callback etc, the ExceptionReceived event handler has to be called. Additionally, if the QueueClient is PeekLock, if the User call back has succeeded and AutoComplete is turned on, the message needs to be completed, else the message will need to be abandoned.

[vinaysurya]

@SeanFeldman, @jtaubensee I've added more details for the OnMessageHandler. Please let me know if you need any more details

[SeanFeldman]

@vinaysurya this is great.

1. When OnMessageOptions.ExceptionReceived event handler is invoked, it doesn't provide information what namespace it's using. Having that information in the new client implementation would help for custom fail-over implementation.

2. Registers a callback is using the underlying infrastructure to receive messages. When shutting down a processing endpoint (process), message receivers and factories are stopped. But that could be right in the middle of message receiving, which can lead to Environment.FailFast() exception. Is there a way to unregister the callback for message processing so that when infrastructure is shut down, it's not affecting any messages in flights?

3. Rate limiting is very difficult with the current OnMessage API. Solution today is to throttle in the callbacks, which causes unnecessary lock expiry and potential dead-lettering.

[vinaysurya]

@SeanFeldman,

1. Sure, we should be able to do this since we have access to the namespace details on the queueclient. But could you please describe more about the failover scenario you mention. Basically today the client automatically retries on exceptions, so even though we surface exception received, the User is not required to take any action and the OnMessagePump carries on with retrying operations etc. So could you please let me know more about the kind of failures you want to react to and the kind of action you are interested in here.
2. Sure, good point. We do not do that today, but will be good to handle with the new client. Simplest thing would be to raise an event when the receiver is closed and have a handler in the OnMessagePump to listen to that close event. This would give a chance for the Pump to do any cleanup action it needs to take when the receiver is closed.
3. I thought today we can use the MaxConcurrent calls to kind of control the rate at which you want to process the messages. Basically the greater this number, the greater the messages we will fetch from the service and invoke the handlers and lesser the number we will make it more and more controlled. Are you suggesting this alone is not sufficient to control the rate of processing from an application standpoint?

[SeanFeldman]

1. Type of failure - namespace is not available or rate of exceptions/errors is high.
2. Whatever the solution that will be, important to take into consideration that receiving of the messages should stop and completion should be alerted. This could get tricky when AutoComplete is set to false and completion is handle in a custom way. Almost as if the receiver would indicate it's about to close and await for the event to be "approved". Only after that to actually close the connection, knowing that there will be no incoming or in-flight messages. awaiting could be set to a timeout of some time to forcefully close connection if "approval" didn't happen.
3. Correct. Please ignore for now.

[vinaysurya]

@SeanFeldman,

1. It looks kind of odd to expose a namespace name on a queueclient api. We could however expose both the namespace name and the queuename as part of the ExceptionReceivedEventArgs as part of the ExceptionReceived handler.
2. Sure, I totally agree that we should give a chance for the OnMessageHandler to perform some cleanup before the client is shutdown(atleast shutdown any further receives). I'm just trying to deliberate how much more elaborate we need to be here and how complex the code ends up as a result. QueueClient shutdown is a scenario that should not be occuring very often, and say when it happens it would result in one abandon of some messages that were in middle of processing/completion. Would that be too costly or are there other consequences that cannot be gracefully handled.

For 2) I am thinking we can start with something simple( say stop further client receives) and then write more test that expose bad failures and iterate to make it better.

[jtaubensee]

PR #75 is in progress

[jtaubensee]

Addressed by #92

[SeanFeldman]

@jtaubensee not quite. These are not addressed yet as #92 is sheer copy of the original OnMessage implementation.

[vinaysurya]

@SeanFeldman, we have tried to address a couple of things differently,

1. Close the messagereceivepump when the client is closed.
2. Also provide the User callback with a CancellationToken, so that their callback is aware if a cancellation has been called. They can check that before calling say a Complete operation and seeing an exception.

[jtaubensee]

Closing based on #92