Nice to have custom retry configuration for checkpointing

[ryancrawcour]

Config option that would allow you to specify a retry of a batch. So if the batch results in an exception, instead of checkpointing, retry the batch.

Set the number of retries. At least 1 retry? At least 5 retries? Unlimited retries?

[ryancrawcour]

@jeffhollan So the default behavior would be to checkpoint as if the batch had completed even if there was a failure in the middle of the batch?

Is this because checkpointing after every message is expensive?

[ryancrawcour]

@jeffhollan what happens if the batch fails even after the retries, do we have some concept of a dead-letter? worried about “losing data” here if we just discard the batch

[jeffhollan]

May help to see how we do in event hubs today. Deadlettering isn’t really a feature of event streams by design. My thinking here is we’d at least have a basic retry policy (e.g. 5xs) and just make sure we pass in the retry count to the active execution. I think we’d have to leave up to user to decide what they want to do if they hit an exception on the 5th time. A bit hesitant to automatically “deadletter” in another stream - at least for the initial version

https://hackernoon.com/reliable-event-processing-in-azure-functions-37054dc2d0fc

[kennetham]

Dead letter queue and circuit breaker pattern should be the 2 patterns we should look at to ensure we have a high scalability stream processing, isn't it?

[jeffhollan]

I don't think we'll pull off circuit breaker at the host level alone as it requires some coordination at the site level which the instance code wouldn't have. As mentioned above I'm a bit hesistant to automatically deadletter, but potentially an optional setting where you can specify a deadletter Kafka stream in your config somewhere and it will deadletter to that for you? But definitely not a must have. In fact this entire issue should be noted as a "should have" or even "nice to have," so ideally we don't spend too many cycles coding for it until other pieces that are more critical are completed.

[ryancrawcour]

Marked as P1 and changed title to "should have" to reflect lower urgency.

[ryancrawcour]

Opened new issue as P2 for "nice to have" to deal with dead lettering

[TsuyoshiUshio]

Close this issue, since we have
#122
Please discuss on it! :)