Stopping a pipeline on error

[slashdotdash]

Error handling is described as follows in the Broadway documentation:

In case of failures, Broadway does its best to keep the failures contained and avoid losing messages. The failed message or batch is acknowledged as failed immediately. For every failure, a log report is also emitted.

Assume there's a requirement that we'd like to manually intervene whenever an error is encountered, instead of allowing processing to continue (due to processing being dependent upon strict message order). Would it be possible to stop a pipeline when encountering an error? Could I use the Broadway.Acknowledger behaviour to signal the producer to stop sending any further messages, or is there a better approach?

Additionally, if the messages were being partitioned it would be preferable to only stop messages within the affected partition, and allow other partitions to continue processing.

I'm interested in using Broadway for event handling (in event sourced applications) and currently support three error strategies: retry, skip, and stop. Retrying messages and skipping (nack) can be handled using the existing Broadway consumer API, but I can't see how to stop processing.

[josevalim]

Broadway was done with the assumption of independent events. Even the idea of partitions is there to guarantee ordering but not to guarantee logical grouping in case of failures. So “stopping a partition” doesn’t make much sense, as you would also stop unrelated events. But I also think what you want can be achieved with a custom producer. The trick is for the producer to hold events for a given partition until the previous element was successfully processed and so on. I believe this is what Kafka consumers do.

…

-- *José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D

[slashdotdash]

Thanks for the feedback @josevalim I’ll look into using a custom producer. Initially it will be simpler just to stop all message processing on error (if requested).

[josevalim]

Perfect, if you have any follow up questions, feel free to post them here! Btw, which source will you be consuming data from?

PS: I am closing it though as there is nothing to do in Broadway, but we can continue the convo.

[slashdotdash]

The messages will be events from Event Store subscriptions (Postgres database).