[ServiceBus] Configuring RetryPolicy in Azure Functions

[APErebus]

I was wondering if it's possible to configure Service Bus retry settings in Azure Functions.

We periodically have a large number of data heavy operations and it would be handy to be able to add a retry falloff so that it doesn't immediately retry if the function fails due to db load or something.

Based on this article: https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus-retry-guidelines
there is the RetryExponential class that would be really helpful, but I can't seem to determine a way to configure it in Azure Functions.

[jeffhollan]

Thanks for bringing this up. I've been looking into this and I need to research more. I think one of the challenges is from what I can tell the RetryPolicy defined here requires the client to keep track of the session and exponential behavior. When having a traditional single receiver this likely works, but for Azure Functions there are no guarantees that the instance that received your first message will be the one that receives the next (or the one that retries). So if a transient exception was thrown in Instance A, and Instance B picked up the message, I don't know how instance B would be aware of when to "try" the message.

Ideally the Service Bus queue would implement state management of retries so the Azure Function host doesn't have to have any knowledge of when to try or retry a queue which gets tricky when multiple instances in parallel.

In the meantime some of the host.json settings for serviceBus may help, as you could limit each instance to a certain number of concurrent calls. Should note though that host.json is for a single instance, and we may scale out to multiple instances as we see. Let me know if that helps and if I'm missing anything on how Retries work. Will get thoughts from Service Bus team as well.

https://docs.microsoft.com/en-us/azure/azure-functions/functions-host-json

[jeffhollan]

Actually on second glance too it looks like this retry policy is only for Service Bus service errors (like if Service Bus was down or being throttled), and not on something like an abandoned message or lock expiration. Either way looking more into this

[APErebus]

Thanks for the response. Looking into the serviceBus host.json settings, it seems that would configure the concurrent calls for an entire functions app? E.g. if we have 10 different functions running services bus triggers, they are all governed by this setting?

Although, as you've mentioned, it appears to be only related to Service Bus errors themselves, rather than failed message completion (due to a user code error or some such).

Looks like we might have to build in a delay on failure.

[jeffhollan]

To answer your question - no it only controls concurrency for a single instance. If you have 10 instances pulling messages you would have 10 x concurrent limit being processed.

[Moeahmad94]

@APErebus @jeffhollan were you able to get a better solution than adding a delay on failure (in an exception handler perhaps)? I'm facing the same issue and am trying to figure out whether there's better feature support for this, or if a manual delay in our own code is the only way for us to do this

[bledbet]

The only real solution I can conceive is to leverage ScheduledEnqueueTimeUtc header, combined with a custom header for tracking how many times you re-scheduled the message - you could use this custom message property (say ErrorCount) to support various backoff strategies (like exponential), and then you'd also need to use it to know when you just need to deadletter the message, since re-sending the message with scheduled enqueue time won't trigger the built-in automatic deadlettering. (In order to deadletter the message yourself, you will need to bind the MessageReceiver that received the message in the function trigger). So the idea is that when your function encounters a transient error, instead of throwing and letting the functions runtime Abandon() the message, you'd access the ErrorCount property of the message (default to 0 if it doesn't exist) and add one - if this is greater than your retry threshold, deadletter it, and if not you'd clone the message, compute your next scheduled enqueue time, set the two mentioned message properties, and send it.

[jeffhollan]

Update: working on a blog and a sample that does this. For now you can check out the code here to do exponential retries with service bus. For now the sample code is here

https://github.com/jeffhollan/functions-csharp-queue-exponential/blob/master/ExponentialRetry.cs

[bledbet]

Update: working on a blog and a sample that does this. For now you can check out the code here to do exponential retries with service bus. For now the sample code is here

https://github.com/jeffhollan/functions-csharp-queue-exponential/blob/master/ExponentialRetry.cs

This looks good - I would recommend also binding the MessageReciever so that when retry-count is >= the specified number of allowed retries, you can use it to deadletter the message.

[jeffhollan]

Great tip. Made the change

[jeffhollan]

Going to close this for now as the plan is to have people implement the retry logic in their code so they can control exactly how and when to retry. The sample pasted above is a great start to this. Let me know if that's not sufficient.