Create a workflow instance idempotent

[saig0]

AT:

• I can provide an unique identifier while creating a workflow instance
• a new instance is only created when no workflow instance with this identifier for this workflow is running

[berndruecker]

Current assumptions:

• It is sufficient to provide uniqueness check "only" for a certain workflow definition
• When we do partitioning of workflow instances of one definition it might get tricky. I asked a prospect for feedback and he commented:

I think this may come down to how you plan to do the partitioning to begin with. Is this down to the client to specify a “partition_key” or do you do that internally ? If it is up to the client, then having deduplication per partition sounds good enough for me as a user can “distribute” the work based on some object_id and thus we are guaranteed to catch the duplicates. The client can supply a specific deduplication_id hash or even reuse the object id and the system will ensure to deduplicate start commands for him ( as far there is something in progress for it ). I have attached a mental map on this scenario.

If you control the partitioning that is a harder problem. At that point a good idea might be to make this functionality configurable per workflow definition and let the user decide if he is willing to incur the extra latency. In such a scenario however having a partion only deduplication sounds inefficient because it pretty non-deterministic as far as the user is concerned.

[berndruecker]

Background motivation can be best explained by a concrete scenario. In an event-driven system based on e.g. Kafka you receive a message by some app which then does something in a database (business logic, transactional) and afterwards sends an event to Kafka that something has been done. If the app crashes after it consumed the event and did the work but did not issue the message - this will not be recognized and something will not happen. Therefor you have to build own infrastructure to "simulate" transactional behavior.
With Zeebe and indempotent start you could leverage a BPMN sequence doing the two things (business logic in DB, emit event) in two steps and be sure both will be carried out).

[saig0]

This will be implemented for message start events in #1012

[MrSaints]

I'm not sure idempotent message start events quite solves idempotent workflow instances.

To elaborate, if I were to create a workflow using CreateWorkflowInstance, I'd either get a success or failure. If I were to publish a message, that will always succeed even if it does not trigger any message start event.

Ideally, it'd be nice to have some confidence when creating a workflow, but also some mechanism to ensure an idempotent instance.

[berndruecker]

Could you share your use case @MrSaints? It is a quite interesting field - but I love moving discussions to a very concrete level, as many things depend on that context. For example I can easily argue, that a duplicate caused by retries or redeliveries can be safely simply ignored - I don't need a feedback here. But there might be use cases where you indeed want hat feedback, and I would love to learn about yours :-)