Running more than one node

You do not have to configure anything to run a workflow application on several nodes. VanillaBP keeps no state in a node. The workflow aggregate, the outbox and the log of processed deliveries live in your database. Every node reads and writes the same rows, and a node which starts can serve any workflow.

What changes with the second node is what each node KNOWS. Three consequences follow from this. The page which owns each topic explains it in full. This page is the list a team works through when it moves to a cluster.

1. The election cache is per node
   What that costs, the one case where it is more than a cost, and when a shared cache is worth writing.
2. A redelivery may reach another node
   Why the delivery record makes that a non-event.
3. A node which did not start a workflow waits differently
   The visibility window of a remote BPMS, seen from a node which knows nothing about that workflow.
4. A node which goes away may leave work behind
   Why no node wakes another, and the one number which decides how late that work goes out.
5. What else the second node changes
   The retention cleanup, the one piece of work only one node does, and the metrics. None of them is a switch.

The election cache is per node

VanillaBP finds out which BPMS (business process management system) holds a workflow. It asks the configured adapters in priority order, and what it learns goes into the election cache. The default cache is a map inside the node. So a second node starts with an empty cache and fills it from its own operations.

For routing this is a cost, but never a risk. An entry is a hint, never an answer. If an adapter no longer knows the workflow, VanillaBP goes back to asking all adapters in order, and the entry is repaired (why). So a cluster of five nodes asks the BPMS about a workflow up to five times instead of once. With an embedded engine this is a query on your own database. With a remote BPMS it is one round trip. If that is the only effect on your setup, you need no shared cache. Run the nodes and let each one build its own.

In one case the empty cache is more than a cost, and you should know it before you meet it. Not everything in the cache was learned by asking. When your application starts a workflow, the node which did it writes down in which BPMS it started that workflow. It knows this without asking anybody, because it did it. This note lets an operation succeed while an eventually consistent BPMS has not caught up yet. The operation is planned, and after your commit the dispatch asks again, for as long as the adapter's window allows. A node which knows nothing about the workflow has no reason to be patient. It asks, and the BPMS answers honestly that it does not know the workflow yet. Your call then raises WorkflowNotFoundException and does not wait.

For correlating a message, for aggregateChanged and for a read of the viewer API, a second note makes this rare. Once the BPMS created the workflow, VanillaBP writes a row about that start into the log of processed deliveries, in your database. Every node reads that row, a restarted node too. It says the same thing the cache would: this BPMS started the workflow. So a node with an empty cache is as patient as the node which started the workflow. The row cannot cover the moment before it exists. It is written after your commit, when the outbox sends the start to the BPMS. So a correlation which reaches another node in the second before that still finds nothing. The row also does not cover a workflow whose adapter does not report the BPMS' own id of it. Nor does it cover a workflow started before the row existed.

That collision needs all three of these at once:

• a remote BPMS which reads from a model that lags behind,
• an operation inside its visibility window (Camunda 8: ten seconds by default),
• and a node which knows nothing about the workflow.

How quickly a workflow finishes does not decide it. What decides it is whether an operation reaches a node which did not start the workflow while that window is still open. The more operations an application moves, the more often this happens. Think of an application which correlates a message right after starting a workflow, and the load balancer sends it to the next node. It may touch the workflow in the second before the start was sent.

The way out is a cache which the nodes share. It is also the answer when asking all adapters in turn becomes the bottleneck. One node writes the association when the workflow starts, and every other node reads it. The BPMS is then asked once per workflow and not once per node.

For Hazelcast this cache is shipped, in hazelcast-shared-election-cache. Add the dependency and say where the other nodes are. The nodes of your application form the cluster themselves, so there is no component to install and none to operate. In Kubernetes they find each other through a headless service, which needs neither multicast nor any address. Read its README to learn how a cluster which does not form looks in the log. A node which finds nobody keeps working, but it shares nothing.

On any other infrastructure you write the bean yourself. This SPI (service provider interface) was written for exactly this case. You write one bean which implements io.vanillabp.integration.spi.WorkflowAdapterCache on whatever the application already runs, for example Redis or a table of your own. The election cache says what it holds and what its methods have to promise. Watching the cache shows how you tell whether yours does its job. On purpose, VanillaBP keeps no registry for you (why).

A redelivery may reach another node

A BPMS hands out work again if it got no answer for it. Nothing makes it choose the node which handled the work the first time. This is a non-event, because of where the record of a processed delivery lives. It is in your database, next to the workflow aggregate, and not in the node which wrote it. Whichever node receives the redelivery reads the same record. It answers from the record and does not run your handler twice.

The same record also says which BPMS holds a task. So a completeTask from any node goes to the right BPMS without asking anybody. Both are described under a repeated delivery.

Two things follow for a cluster. First, an application which switched the deduplication off (deduplicate-deliveries: false) handles the repetition in its handlers. On several nodes this means handlers which are safe to run twice, on two different JVMs. Second, the retention of the records (vanillabp.delivery.retention) is a property of the store, not of a node. So raising it affects the whole cluster, and one cleanup deletes by age across the whole table.

A node which did not start a workflow waits differently

A remote BPMS answers questions from a read model which lags behind its own writes. VanillaBP knows this and waits where it can. An operation on a workflow which this node started is planned and retried until the BPMS admits that the workflow exists. The adapter's own window limits this. For Camunda 8 it is workflow-visibility-timeout, ten seconds by default. Camunda 7 has no lag at all. What the engine behind the Process-Engine-API does is up to that engine, see choosing a BPMS.

Now look at it from a node which neither started that workflow nor received a delivery for it. For this node there is practically no window. The patience comes from the record in the cache, not from the clock. So a node without the record does not wait a shorter time. It does not wait at all. For an operation on the workflow itself, the row written when the workflow started takes the place of the record. It is in your database, and every node reads it. So the two levers are really one: share the cache, and a node's patience no longer depends on which node started the workflow. Adapters which can only guess says everything an adapter can and cannot be asked while its BPMS catches up.

A node which goes away may leave work behind

Nothing tells another node about an outbox entry. This is on purpose, not something missing. The node which writes an entry is awake by definition, and it sends the entry itself, right after its own commit. A node which sleeps through that write loses nothing, because nobody waited for that node in particular. Every node can serve every workflow. So a cluster needs no notification between its nodes. A database notification such as Postgres LISTEN/NOTIFY would gain nothing. And it would cost the portability VanillaBP promises across the databases it supports and across MongoDB.

What remains is a node which goes AWAY between writing work down and doing it. This can be an entry it inserted, a retry it rescheduled, or an entry it was just sending. Meanwhile the other nodes sleep until a time which they computed before that entry existed. A crash and a graceful shutdown are the same case here. Stopping ends the poller and leaves behind whatever it had not sent yet.

An entry which a node took carries the claim of that node. The claim keeps the other nodes away from it from the moment it was taken until the dispatch is over. That holds however long it takes, because the node which holds the claim keeps moving the moment forward. An entry which still waits for its turn is held the same way. A node which disappears stops moving it forward. So the claim runs out one vanillabp.outbox.attempt-frequency after its last sign of life, and the next poll of any node takes the entry over. It does not matter which node that is. The operation reaches the BPMS once and not twice, because no two nodes ever hold one entry at the same moment.

vanillabp.outbox.poll-interval covers this case, and nothing else. It is the longest time a poller sleeps while its own store owes nothing, ten seconds by default. Every VanillaBP application polled at this rhythm before the sleeping existed. If you raise it to minutes, a cluster stops paying for polls it does not need. The only cost is how soon the remaining nodes pick up the work of a node which died. Nothing else gets later: an entry with a due time is sent at that due time, whatever the cap is.

So the sizing question is a short one. How long may the work of one lost node wait before somebody notices? That is your cap. It is not the answer to how late an ordinary entry may go out. Reading it that way is the one mistake which gives the whole saving back.

What else the second node changes

The retention cleanup of the delivery records runs in every node. It deletes by age across the whole table, not by anything a node owns. So two nodes which clean up at the same moment remove the same expired rows and nothing else. How long a delivery is remembered says what the period has to cover. The part nobody can compute for you gets longer with a rolling deployment, because a node which is down refreshes nothing.

The housekeeping of the outbox is the one thing which only ONE node does. In a window at night, it removes the dispatched entries whose retention has passed and the payloads which no entry names. It works out how much fits by measuring what it does. Two nodes doing this at once would each measure the other's work. So a node writes a claim into the store's own database, in the table VANILLABP_HOUSEKEEPING or the collection vanillabp-housekeeping. The other nodes leave that store alone until the window closes. The claim is per STORE. So in an application in a migration, one node may do the housekeeping of its JPA outbox and another node that of its MongoDB outbox. There are two consequences for a cluster. A node which dies inside the window holds its store until the window ends, and this costs the rest of that night. And a cluster which is scaled to zero at four in the morning does no housekeeping at all that night. Nothing catches up on either of them. The three gauges on whether the housekeeping window is wide enough show you a missed night.

The metrics are per node, as a scrape expects. A counter for the whole cluster is the sum your monitoring builds. VanillaBP does not add them up. So write an alert on the counters worth watching against that sum.