Explanation of the built-in Kafka Retry Policy

[donntorres35]

Hello, I'm new to Kafka integration and I'm tasked to read/write to a Kafka topic using Azure Functions. My concern is the retry policy for both consumer and producer... can anyone explain the internal mechanisms how these policies work? If a certain message consumed/produced reached max retries, where does it go? Would a topic named "my-topic-dead-letter" appear and I can have a custom processor read or use that to send notifications or do I have any options in-code to check if a message is at max retry and handle accordingly?

TIA

[shrohilla]

Hi @donntorres35,
To answer your question :-

1. internal mechanisms how these policies work?
   Based on the configuration, we retry the function internally and if the limit is breached Kafka Extension use to mark that event commit and move ahead. This support has other major benefits like replay those events which are failed due to timeout on downstream services or any other network lag happened, in this case the complete customer function runs from the start so if any persistence is there customer function should manage that

2. Would a topic named "my-topic-dead-letter" appear
   Kafka extension currently don't support any dead letter queue concept out of the box.

Customers are recommended to add the dead letter topic in their respective Kafka broker and use the output binding to write in dead letter topic and replay those events based on business use-case.

[donntorres35]

Hello @shrohilla, thanks for your response. Good insight on bullet 2 as that will align our expectations on dlq custom build and processing. Additional question, the Kafka Confluent software will be hosted on GCP, any insights if the Azure Functions bindings will still work if it's in GCP and not in Azure as long as it's in the same region?

Thanks in advance

[shrohilla]

HI @donntorres35 ,
Not sure what exactly is your question

• Kafka extension is the part of Azure Functions so it will run directly on Azure.
• Kafka Broker you can run on any cloud version but we majorly supports managed Kafka Broker like Confluent Cloud or EventHub
• Not sure how Confluent Cloud running with GCP, if GCP is the interface between Confluent Cloud and customer than answer is yes