BPMS migration

VanillaBP can run one application against more than one BPMS (business process management system) at the same time. So you can move from one BPMS to another without touching business code. That works from an on-premises cluster to SaaS, from an old engine version to a new one, or from one vendor to another.

1. How it works
   The configuration which is the whole feature, and what happens to new and to existing workflows.
2. Election guarantees
   What VanillaBP promises about which BPMS an operation reaches.
3. Adapters which can only guess
   The one property of a BPMS which decides whether a migration works.
4. The election cache
   What VanillaBP remembers about a successful election, and why losing it costs only time.
5. When a workflow ends
   What happens to a cached election when its workflow is over.
6. Watching the cache
   The meters the cache publishes, and what a rising number means.
7. Naming adapter ids, and never renaming them
   Why VanillaBP stores an id, and what to do instead of renaming one.
8. Two adapter ids on one system
   Why every adapter id needs a system of its own, and how VanillaBP checks it at startup.
9. Consistency of workflow starts (remote BPMS)
   What the two-phase start guarantees while several BPMS are configured.
10. Leaving tenants behind
    How to change the name-clash-avoidance mode with a migration instead of a setting.

How it works

Configure every BPMS as an adapter instance and put them in order of priority:

vanillabp:
  prioritized-adapters:
    - new-bpms          # first priority: new workflows start here
    - old-bpms          # existing workflows are still found here
  adapters:
    new-bpms:
      type: camunda8    # see 'Available BPMS adapters' for the types available
      # ... connection settings of the new cluster
    old-bpms:
      type: camunda7

The type selects one of the available BPMS adapters. The adapter's own wiki documents its connection settings. You may configure the same BPMS type under several ids (e.g. two clusters of the same product). You can also set the priority list per workflow module and per workflow (vanillabp.workflow-modules.<module>.prioritized-adapters / ...workflows.<bpmn-process-id>.prioritized-adapters). Which of them counts is explained in the levels a setting is written at.

Two rules decide where each operation goes:

1. New workflows always start in the adapter with the first priority. VanillaBP does not ask any BPMS on start.
2. Operations on existing workflows go to the BPMS which holds the workflow. Completing or canceling an asynchronous task, completing a user task and correlating a message all elect the adapter. VanillaBP asks each adapter, in order of priority: "do you know this workflow/task?" The first adapter which answers "yes" runs the operation.

So a migration is only a change of configuration. Move the new BPMS to the first priority and keep the old one in the list. New workflows start in the new BPMS. Existing workflows run to their end in the old one. When the old BPMS runs no more workflows, remove it from the configuration.

VanillaBP deploys the BPMN models to all configured adapters of a workflow module. That is the union of the priority lists on module level and on workflow level. So every listed BPMS is ready to serve its workflows.

Election guarantees

• An adapter answers only for the workflow it is asked about. The question carries the workflow module and its BPMN processes, not only the aggregate id. This matters for two reasons. First, aggregate ids are unique per aggregate type, not across your application. So two workflow modules can both hold an id 1. Second, two adapters may share one backend. That is what a migration between two scopings of one BPMS looks like, and there a task id can be reached from both sides. Each adapter translates the question into what its BPMS knows the processes by (a tenant, prefixed identifiers, a table prefix). For everything else it answers "unknown". VanillaBP's adapters do that. An adapter of your own has to do it too (MigratableProcessService, section "The election contract").
• The same holds for what an operation WRITES, not only for what a probe answers. An example is a changed aggregate which is pushed into a workflow. On an embedded engine, this write makes conditional events evaluate again. Writing it into a workflow of another module or of another adapter id would move on a workflow nobody asked about. You do not have to do anything for this. It is named here so that a reader who checks whether their setup is safe does not need to know which version fixed which of the two paths.
• If an adapter answers "the workflow is unknown to me", the next adapter in the list answers.
• If an adapter cannot be reached, the operation stops. VanillaBP retries the probe for a short time and then fails the business operation. It never falls back to the next adapter. The BPMS which cannot be reached might be the one which holds the workflow, and running the operation somewhere else would be wrong. Retry the business operation when the BPMS can be reached again.
• If the workflow is reported as already completed, the operation does nothing (logged as a warning).
• If no adapter knows the subject, an exception explains the likely causes and guides you (TaskNotFoundException / WorkflowNotFoundException). (why)
• Sometimes VanillaBP knows the id the BPMS uses for the workflow. Then it sends that id along with the question, on the paths which have to wait for an answer. The id is a shortcut, never the answer. An adapter may use it to say "yes" earlier than its read model can. It must never take "the engine does not know this key" as "the workflow is unknown to me". An engine forgets a workflow as soon as it ends, and only "unknown" sends the next operation to another BPMS. VanillaBP does not ask any BPMS for this id. It takes it from the record VanillaBP wrote when a task of that workflow was delivered, and from the election cache.

Adapters which can only guess

An adapter answers the election from what it can ask its BPMS. In two configurations it cannot ask, so the answer is an optimistic yes:

• The Process-Engine-API adapter, always. That API delivers tasks, but it answers no question about a running instance.
• A Camunda 8 adapter which booted DEGRADED. That adapter requires a cluster it can search. Normally a cluster which refuses ends the boot. The boot only goes on for an adapter which is not the module's adapter with the first priority and which has deployment-failure: warn. Such an adapter deployed nothing, so it cannot find a workflow either. That is exactly the migration AWAY from such a cluster, and this is the only reason why this case exists.

Such an adapter wins every election it takes part in, because the walk stops at the first ACTIVE. With one BPMS configured, that costs nothing. This is why the configuration is allowed at all: the only candidate is the right one. In a migration, it decides everything. Say you have two adapters in prioritized-adapters, and the first one guesses. Then every operation goes to that first one, no matter which BPMS really holds the workflow. The second one is never asked. The setup then routes by the order of the list instead of by knowledge. Only the missing progress shows it.

VanillaBP does not let you start such an application. Each adapter says whether it can ask its BPMS at all. If a workflow module puts a guessing adapter in its priority list next to another one, the boot ends. The message names the adapter, the module and the ways out. The check runs after the adapters deployed, because only then does an adapter know what its BPMS can do. It also runs before anything touches a workflow. (why)

What to do about it, in this order. First, give that BPMS what it needs to answer exactly. On Camunda 8 that means a cluster the adapter can search. Second, if the guessing adapter is the OLD BPMS of the migration and none of its workflows is left, remove it from the list. If the new BPMS of a migration can only guess, postpone the migration. There is one case where you accept the routing below as the answer and not as a shortcut: a migration away from a Camunda 8 cluster which cannot be searched. You cannot give the old adapter what it needs, and that is why you are moving away from it.

You may want that routing anyway, for example in a test setup or in a migration you steer by hand. Then say so, and the message becomes a warning instead of the end of the boot:

vanillabp:
  election:
    guessing-adapters: ACCEPTED     # or per module:
  workflow-modules:
    my-module:
      election:
        guessing-adapters: ACCEPTED

The election cache

VanillaBP caches successful elections (workflow module, BPMN process, aggregate ID → adapter ID). So repeated operations on the same workflow skip the probing walk and ask the right BPMS directly. The default cache lives in one instance of your application. What that means for several instances is collected under running more than one node. Cache entries are hints, not the truth. If a cached adapter no longer knows the workflow, VanillaBP falls back to the full probing walk and repairs the entry. A stale entry costs one more probe, but never a wrong result. (why)

Elections are not the only thing which fills the cache. VanillaBP also records what it knows without asking anybody. That is after it started a workflow in a BPMS, and whenever a BPMS delivers something of that workflow (a task, a user task, a start the BPMS did itself). VanillaBP records the end of a workflow too, but in a different way. See When a workflow ends.

This record lets an operation succeed shortly after a start, also on a BPMS which answers from an eventually consistent read model. Such a BPMS honestly reports a workflow it created a moment ago as unknown. VanillaBP knows better, because it started that workflow there. So it plans the operation and returns. The dispatch runs after your commit and asks again, for as long as that BPMS says it may need (Camunda 8: workflow-visibility-timeout, 10 seconds by default). VanillaBP does not wait for a workflow which nobody recorded. A wrong ID raises WorkflowNotFoundException in your call, as it should.

Nothing waits inside your transaction. The election runs where you called correlateMessage, completeTask or aggregateChanged. There it asks each BPMS once and then gets out of the way. Your transaction commits, its database connection goes back to the pool, and the rows it wrote are unlocked. The waiting happens where the entry is dispatched. That is waiting for a BPMS which cannot be reached, and waiting for a read model which has not caught up. There, it costs the entry time on the store, and it does not cost your application a connection. (why)

Reading is the exception, for the same reason. getWorkflowHistory and getProcessDefinitions have nothing to plan, and nobody repeats them later. So they wait themselves, limited by the same record and the same time window.

What you see in each case:

• A BPMS which reports itself as not reachable fails your call at once. The message names the adapter.
• A task the BPMS does not know fails at once as well. That answer is exact, and asking again would change nothing.
• Your application started a workflow, but the BPMS has not made it searchable yet. This costs nobody's time. The operation is planned, your caller commits, and nothing waits for the cluster. The entry is due again after the time window which the adapter names for its read model. On Camunda 8 those are the ten seconds of workflow-visibility-timeout. Lower that value if your cluster catches up faster and you want the operation sooner. This kind of waiting uses none of the attempts of vanillabp.outbox.block-after-attempts.
• If the exporter of the cluster stops, the entry is repeated until vanillabp.outbox.wait-for-visibility-at-most has passed since it was written. That is about four hours by default. Only then is the entry blocked. If the work hangs on a job, the cluster raises its own incident.

The record in the cache is not the only thing which says that your application started a workflow. After the BPMS created it, VanillaBP also writes a row about that start into the log of processed deliveries, in your database. Correlating a message and aggregateChanged read that row when the cache has no record. So they stay patient after a restart, on another node and after the cache forgot. Before, such an operation failed with WorkflowNotFoundException while the read model was behind. And an operation which was already planned was dropped as stale when it was dispatched.

The default cache is in memory, has a size limit and expires entries. You do not have to configure anything. If your application keeps more workflows active than the cache holds, raise its limits:

Property	Default	Meaning
vanillabp.workflow-adapter-cache.max-entries	10 000	how many records are kept; above that, the one used least recently is dropped
vanillabp.workflow-adapter-cache.time-to-live	PT1H	how long a record is kept
vanillabp.workflow-adapter-cache.ended-time-to-live	PT5M	how long the record of a workflow which ended is kept
vanillabp.workflow-adapter-cache.release-on-workflow-end	false	whether the BPMS reports the end of a workflow, so its record can go early

A record takes about 300 bytes. So the default cache fills about 3 MB, and 100 000 records fill about 30 MB. The limit is strict on purpose: a lost record costs one more round of asking the BPMS, never a wrong answer.

Without a shared cache, several nodes leave a small gap. The record in the cache is held by one node: the node which started the workflow or received a delivery of it. The start row described above covers most of the rest, because every node reads it. Three cases are left. The row is written only after your commit, when the outbox sends the start to the BPMS, so an operation in the moment before has nothing to read. An adapter which does not report the BPMS' own id of a workflow writes no such row. And a workflow started before the row existed has none. In these cases an operation which reaches another node does not wait. It fails with a WorkflowNotFoundException while the BPMS is still catching up. Retrying the business operation works, because by then the workflow is searchable. A shared cache closes this gap. Running more than one node says when that is worth it.

In a cluster, all instances of your application can share elections through one bean which implements io.vanillabp.integration.spi.WorkflowAdapterCache. That bean replaces the default automatically. On Spring Boot this happens by bean overriding. On Quarkus it happens because the default is a @DefaultBean.

For Hazelcast, the bean comes ready-made and needs no code at all. You only add the dependency io.vanillabp:hazelcast-shared-election-cache-spring-boot or ...-quarkus, plus the settings which say where the other nodes are (hazelcast-shared-election-cache). The nodes of your application form the cluster themselves. In Kubernetes they find each other through a headless service. This needs no multicast, and you configure no addresses.

On any other infrastructure you write the bean yourself. It has three methods (get, put, invalidate), plus putEnded if you want the shorter lifetime of the next section. Use Redis, a table of your own, or whatever the application already runs. This interface is the contract, and a cache written against it is not a second-class solution.

A shared cache also helps when the probing itself gets expensive. VanillaBP asks the BPMS which adapter holds a workflow. It does not keep a register of its own. With an in-memory cache, every node asks that question once per workflow. With a shared cache, the node which started the workflow writes the association and every other node reads it. So the BPMS is asked once per workflow, not once per workflow and node. (why) Nothing else has to change, and the entries stay hints. A node which reads one still probes the adapter it names. So a wrong or stale entry costs a walk, but never a wrong route.

A call which names a task needs neither. This covers completeTask, cancelTask and an aggregateChanged which names the task whose scope receives the values. VanillaBP answers them from the delivery record of that task. That record lives in the database of the workflow aggregate. So every node reads it without a cache and without asking any BPMS. See The record also says which BPMS holds the task. (why) The advice above still holds for everything else: correlating a message, a push to the whole workflow, and reading the viewer API. It also stays the answer wherever no record exists.

When a workflow ends

A workflow which ended has a record in the cache, and that record can never be useful again. But VanillaBP does not delete it, because "the workflow is over" does not mean "nothing else arrives". These can all still show up: a completeTask which lost its race against a timeout, a message correlated by an endpoint which did not learn about the end, an outbox entry dispatched after the end, and a read of the viewer API. With the record, each of them asks the BPMS which held the workflow. It hears that the workflow is completed and becomes the no-op with a warning described above. Without the record, the full walk runs. When the BPMS has forgotten the instance, that no-op turns into a WorkflowNotFoundException. (Camunda 7 forgets after its history-time-to-live. Camunda 8 forgets when the secondary storage drops the finished process.)

So VanillaBP MARKS the record instead of deleting it. The record still answers, and it is kept for ended-time-to-live (five minutes) instead of time-to-live (one hour). This makes the difference in a shared cache. Think of a cluster of a dozen nodes which runs thousands of workflows a day. It pays for the records of workflows which ended long ago, and this setting takes them out.

VanillaBP knows that a workflow ended only if the BPMS says so. And the BPMS says so only if somebody asked. Three things ask: a @WorkflowEnded method of your application, vanillabp.delivery.release-on-workflow-end, and vanillabp.workflow-adapter-cache.release-on-workflow-end. Each of them attaches a listener or a worker to every deployed process of the workflow module. One of them costs as much as all three. So the cache costs nothing extra if you already use one of the others. That is why the switch defaults to false. An application which never asked about the end of its workflows should not start to pay for it because of a cache.

It stays best effort. The Process-Engine-API does not report the end of a workflow at all. Camunda 8 reports a completed workflow, but never a terminated one. Where no notification arrives, the record lives its normal time-to-live, as it did before.

Your own cache bean decides for itself. putEnded is a default method which falls back to put. So a bean written against the three methods it had before keeps working without change, and its records of ended workflows live as long as any other. Implement the method to give them their own lifetime, the way your infrastructure expresses it (Redis: its own EX; Hazelcast: a per-entry ttl).

Watching the cache

Your application may use Micrometer. On Spring Boot, the Actuator exposes whatever Micrometer knows. On Quarkus, the Micrometer extension does. Then VanillaBP reports these meters.

These three say what the election asked. They are reported for every cache: the default, the one VanillaBP ships for Hazelcast, or one you wrote yourself.

Meter	Meaning
vanillabp.workflow.adapter.cache.hits	operations answered from the cache
vanillabp.workflow.adapter.cache.misses	operations which had to ask the BPMS
vanillabp.workflow.adapter.cache.ended.marks	records of workflows which ended, since the application started

These five belong to the default cache, because only it can know them. They carry the name of that implementation. A different cache publishes what IT knows under its own prefix instead:

Meter	Meaning
vanillabp.inmemory.election.cache.size	records held right now
vanillabp.inmemory.election.cache.size.ended	how many of the records held belong to ended workflows
vanillabp.inmemory.election.cache.evictions	records dropped because the cache was full
vanillabp.inmemory.election.cache.evictions.unused	records dropped before they were ever used
vanillabp.inmemory.election.cache.lost.hints	operations which would have been a hit with a bigger cache

Without Micrometer, none of this is published, and your application runs as before. With a shared cache, the second table is simply missing instead of reporting zeros. A meter which cannot mean anything is worse than a missing one, because a dashboard shows it as a broken cache.

size.ended tells you whether the release works. It rises while workflows end and falls again when the shorter lifetime removes those records. If the number stays at zero while workflows end, the end is not reported to VanillaBP. Then look at the switch of the previous section. With a cache which does not report its size, ended.marks from the first table answers the same question.

The last row shows whether the cache is too small. A full cache does not show it, because a full cache is normal. A record of a workflow nobody asks about again is no loss. When ten records were dropped for lack of space and then looked up again within an hour, VanillaBP logs a warning. It names the number and the property to raise. A healthy application never sees it.

Naming adapter ids, and never renaming them

An adapter id is the name of one BPMS instance in your configuration. Choose it with care. An id without a type is taken FOR the type. That is a convenience for applications which upgrade from VanillaBP 1, where there was one adapter and its id was its type. It is not a recommendation:

• An id says which system it is, not which product. Use camunda7-frankfurt and camunda7-vienna if two engines run in two datacenters. Use camunda7-loans and camunda7-claims if they differ by the database they own. Use camunda8-saas next to camunda8-onprem, or camunda8-8-7 next to camunda8-8-9 while you leave a cluster version behind. An operator who reads a log line, a metric tag or a viewer's process definition id sees the id, not your configuration.
• A migration needs two names anyway. Say you move from one Camunda 8 configuration to another one: another cluster, another scoping or another cluster version. Then both configurations run side by side as long as the old workflows take. They cannot both be called camunda8. If you choose a meaningful name from the start, you avoid the one rename you must not do (see below).
• Only the type has to be named where the id is not the type: type: camunda8 in the section. Nothing else changes.

Caution

An adapter id must never be renamed while workflows are running. VanillaBP uses it to know what belongs to which BPMS, and it stores the id. Renaming an id is not a rename. It creates a new adapter which never processed anything, next to an old one which nothing serves any more.

Where the id is stored, and what a rename costs:

Where	What a rename does
the phase-two outbox, for every open START entry	the entry names the adapter elected in phase one. Under the new name nothing serves it. So the dispatch fails with a message which guides you, and in the end the entry is blocked. The workflow was stored but never started
the delivery log of every task processed so far	the delivery key is built from the delivering adapter, the workflow module, the BPMN process, the event and the delivery id. Under a new name every key changes. So a repeated delivery no longer finds its record: the @WorkflowTask method runs a second time, and the age of an open task starts again from zero
the election cache	entries become useless hints. This does no harm: the election falls back to the walk and repairs them
process definition ids handed to the application	the viewer's ids contain the adapter id as a namespace (<adapter-id>#<native-id>). So if a UI still holds an old id, it gets ProcessDefinitionNotFoundException until it asks again

So the way to change a BPMS is always the same, and it is the recipe of this page. Add the new adapter id and put it first in prioritized-adapters. Let the old workflows finish in the old id. Remove the old id when nothing runs there any more. Removing an id whose workflows have ended costs nothing. Renaming one costs the workflows. (why)

VanillaBP notices the leftovers at startup. While the application boots, VanillaBP asks its own stores, once per BPMN process, for which adapter ids they still hold something. That means the outbox entries which are waiting, and the records of tasks which are still open. If an id is no longer configured, VanillaBP logs a WARN which names both possible causes. Both cost something, and only the application knows which one it is:

The adapter id 'camunda8-old' is NOT configured any more, but it still has waiting phase-two outbox
entries of BPMN process 'LoanApproval' (workflow module 'loan-approval'). [...] the id was RENAMED -
then the workflow was persisted and never started - or it was removed while its BPMS still had work
left. To solve this either
  - configure 'vanillabp.adapters.camunda8-old.*' again [...], or
  - state that you know about the leftovers: vanillabp.retired-adapters: [camunda8-old]

It warns and does not fail the boot. The entries are not lost, they wait. And an application which is being repaired must still be able to start. When the last step of a migration is done and you know about the leftovers, say so once:

vanillabp:
  retired-adapters:
    - camunda8-old      # its workflows ended; whatever is left in the stores is known

(why)

One limit remains in every setup. An entry which used up vanillabp.outbox.block-after-attempts is blocked and no longer part of the question. It waits for a person, not for a dispatch, and it has its own ERROR which names the workflow it lost. A rename while nothing is waiting and no record is left cannot be detected at all. There is nothing to compare against. But nothing breaks either, except process definition ids which an application already holds.

Two adapter ids on one system

Each adapter id needs a system of its own. Two ids on one system are one BPMS with two names. Both would work on the same data and get in each other's way. VanillaBP checks this at startup, before anything is deployed, and the start ends if two ids share a system.

• Ids of the same type are compared by their adapter. Only the adapter knows what makes two of its instances different. For Camunda 7 this is the datasource and the table prefix. The page of each adapter says what it compares.
• Ids of different types are compared by VanillaBP itself. Some adapter types run on the same kind of system. Camunda 7 and its forks CIB seven and Operaton all use the same database tables (ACT_*). Such an adapter tells VanillaBP which system an id works on. VanillaBP refuses two ids with the same system, whatever their types are.

The second check matters exactly when you migrate. Say you move from Camunda 7 to one of its forks and run both engines side by side. If both engines use the same datasource without a table prefix, each job executor runs the jobs of the other one. Nothing fails at startup, and the damage shows later. So VanillaBP stops the start and says what to do:

The adapters 'c7' (type 'camunda7'), 'cib' (type 'cibseven') work on the SAME system: datasource
<the application's default datasource>, table prefix <no table prefix>. [...] Give each adapter id
a system of its own:
  - for type 'camunda7', 'cibseven': give each one its own datasource
    ('vanillabp.adapters.<id>.data-source-name'), or its own engine tables on the shared datasource
    ('vanillabp.adapters.<id>.table-prefix', [...])
or remove all but one of these adapters.

Ids of different types on different systems are fine. That is the migration setup.

Adapters whose types can never meet, such as Camunda 7 and Camunda 8, do not take part in the second check. (why)

Consistency of workflow starts (remote BPMS)

Starting workflows in a remote BPMS uses a transaction outbox. The start is stored with your local transaction and runs against the BPMS after the commit. If the application crashes in between, VanillaBP dispatches the entry again. This gives at-least-once semantics: in rare crash cases the same start may reach the BPMS twice. VanillaBP keeps duplicates low on several layers. There is one unique outbox entry per workflow, dispatched entries are kept, and before every RE-dispatch a probe checks whether the workflow exists by now. Still, a small window remains, as a property of eventual consistency. See your adapter's wiki for the details of your BPMS.

Leaving tenants behind

The same mechanism moves a setup from one name-clash-avoidance mode to another. The practical reason: BPMS vendors charge licenses per tenant. So you may save money if you leave tenants behind and use prefixed identifiers instead. This cannot be a simple change of a property. The identifiers the BPMS knows would change, and workflows started earlier would no longer be found. So migrate instead:

vanillabp:
  prioritized-adapters:
    - camunda8-prefixed      # new workflows start here
    - camunda8-tenants       # workflows started earlier still live here
  adapters:
    camunda8-prefixed:
      type: camunda8
      rest-address: http://localhost:8080      # the SAME cluster
      name-clash-avoidance: use-prefix
    camunda8-tenants:
      type: camunda8
      rest-address: http://localhost:8080
      name-clash-avoidance: by-adapter

Both schemes live side by side on one cluster. That is exactly what the prefix gives you: a prefixed deployment cannot clash with one based on tenants. New workflows start in the adapter with the first priority. Operations on existing workflows go to the adapter which holds them. When no workflow of the old scheme is left, remove the old adapter id. Then you can delete the old tenant in the BPMS.

Two adapter ids of one BPMS type need something from that BPMS, and it differs per BPMS. One BPMS may need a way to tell which of the two ids a key belongs to. Another may need separate storage for its two engines. A third may refuse the setup. In every case, the boot tells you with a message which names both ids. It does not let the two ids overwrite each other. Before you plan such a migration, read what your BPMS needs: Camunda 8, Camunda 7. A single adapter id of a type needs none of this.