New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event Hub-triggered function *always* checkpoints -- I want to control checkpointing #947

Open
vfab opened this Issue Sep 11, 2018 · 0 comments

Comments

Projects
None yet
1 participant
@vfab

vfab commented Sep 11, 2018

I've already posted this on Azure Advisors but no response from Microsoft there. Retrying here...

This document (https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs) states plainly that at the end of execution of an Event Hub-triggered function, the function will checkpoint whether there was an error or not. (I believe WebJobs have the same behavior.) Unfortunately this doesn't give us enough control. Perhaps there was a throttling error, or some other condition that means the messages can't be processed successfully. In such cases I would like to be able to tell the EventProcessorHost not to checkpoint.
I consider being able to control checkpointing as a must-have feature if you're processing events from Event Hubs (or IoT Hub). Without this control, Event Hub-triggered functions (and WebJobs) are not sufficiently reliable, because it's obvious that you can lose messages in error scenarios. Cloud-native apps are supposed to handle failures gracefully, but that's not the case here.

Just to be completely sure about this, I wrote a function to test this out. It reads in batches of messages and throws an exception at the end. Upon launching the function, it happily reads through all of the messages in the Event Hub, checkpointing every batch. So I'm quite confident that the EventProcessorHost DOES checkpoint even if there is an exception.

So is there a way to control checkpointing currently? (And I don't mean messing with checkpoint blobs.) I don't think there is, and if that's truly the case, then is it possible that you could add a feature to tell the EventProcessorHost whether or not to checkpoint?

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment