Provide a configurable retry policy for Kafka triggered functions

[mcollier]

As a developer,
I would like the ability to optionally configure a retry policy when the triggered function fails,
so that a failed execution does not automatically advance the cursor.

This feature should work for all GA supported Azure Functions languages.

There is a similar issue for the Event Hub extension. The expectation is that the Event Hub and Kafka extension work in a similar manner related to retries. Please see the discussion at Azure/azure-webjobs-sdk#1597 and the sample at https://github.com/jeffhollan/retry-design.

[maxbog]

Hi @mcollier. Is this something that is being worked at?

[mcollier]

@maxbog I don't have any insights on specific work for the Kafka extension.

FYI - There is now a retry policy for function triggers; see https://docs.microsoft.com/azure/azure-functions/functions-bindings-error-pages?tabs=csharp#retry-policies.

[maxbog]

@mcollier I saw those annotations, but from this issue I understand that they need some code in the kafka trigger to be supported, right? What I'm looking for is a retry on an exception without advancing the cursor.

As far as I know, right now even if an exception occurs in the function code, the kafka cursor will go forward.

Maybe I could help here and submit a PR based on how it's implemented in the Event Hubs trigger? I saw a PR on the EH repo, which is not merged yet: Azure/azure-functions-eventhubs-extension#56

[maxbog]

I just read some code in the Azure/azure-webjobs-sdk#2463 PR and for me it seems that there is nothing required on the kafka extension to support the retries.

@mcollier Can you confirm that the Retry attributes will work without any changes to this repo?

[TsuyoshiUshio]

Hi @maxbog

I talked with the author of the Azure/azure-functions-eventhubs-extension#56 . He said that the PR should be closed since he imiplemented it on webjobssdk side. We can use the retry feature for the Kafka as well. I'll test it and confirm if it works. However, in theory, it should work. :)

[TsuyoshiUshio]

Hi @maxbog and @mcollier
It works for Kafka Trigger as well, however, the issue is, the retry is supported on Microsoft.Azure.WebJobs 3.0.23. I need to update the version. I'll do it soon. Before that, you can add WebJobs on your csproj file. I'll upgrade the kafka version to fit 3.0.23 soon.

     <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Kafka" Version="3.1.2" />
     <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.10" />
    <PackageReference Include="Microsoft.Azure.WebJobs" Version="3.0.23" />

[TsuyoshiUshio]

Hi @maxbog @mcollier
I have an update. The Microsoft.NET.Sdk.Functions is updated. It includes WebJobs 3.0.23 So we can go with

     <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Kafka" Version="3.1.2" />
     <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.11" />

[paulbatum]

Retry policies now require each trigger to opt-in (because many triggers already have built-in retries) so there is still some work to do here for the Kafka extension to add retry support.

[shrohilla]

This is supported in 3.5.0 release