Spring Boot integration

This section explains the minimum you need to know to get VanillaBP running in a Spring Boot project: structure and dependencies, no theory.

1. Choose an appropriate project structure
   VanillaBP works for monolithic applications and for microservices. It scales from small projects with one module to large Maven or Gradle builds with many modules. Your setup decides which dependencies you need.
2. Add the required dependencies to the correct Maven/Gradle modules
   VanillaBP has a small set of dependencies, each with a clear purpose. Add each of them where it belongs.
3. Configure the transaction outbox
   Only needed for adapters of a remote BPMS (business process management system). VanillaBP uses a transaction outbox to start workflows reliably. It works without any setup, and you can tune it with the vanillabp.outbox.* properties.
4. What VanillaBP remembers about delivered tasks
   The delivery log, which answers a repeated delivery instead of running your handler again, and the little it needs from you.
5. Creating the tables with Liquibase or Flyway
   The four tables VanillaBP owns, written out for a project which manages its schema itself, and what the same switch means on MongoDB.
6. Configure the adapters
   Each BPMS you use gets a section below vanillabp.adapters. Optional settings decide what happens when a deployment fails, and how to cope with a BPMS which is not available for a while.

Choose an appropriate project structure

VanillaBP supports business process applications built as a monolith or as microservices. Both work with a single module as well as with large Maven or Gradle projects with many modules.

If you use several modules, you usually keep the workflow modules apart from the main Spring Boot application. The workflow modules page has concrete examples.

To decouple them further, each workflow module can bring its own configuration files (YAML and/or .properties), named after the workflow module ID. VanillaBP loads these files automatically and merges them into the Spring Environment as defaults. Whatever the application configures wins over them, application.yaml and application-prod.yaml included. Profile-specific files (e.g. loan-approval-prod.yaml) work as well. In projects with several modules, you can put the configuration files into a subdirectory named after the workflow module ID (e.g. loan-approval/loan-approval.yaml). That is the same convention as for all other resources of a workflow module.

The main rule: put everything a workflow module needs into its own configuration files. The application.yaml should only hold values which really concern the whole application, such as database settings, server configuration or the definitions of the VanillaBP adapters. Properties of one module, like endpoint URLs, timeouts or feature flags, belong in the module's own files. So the workflow module more and more becomes a self-contained application inside the application. See Configuration for the details and for which file wins.

Add the required dependencies to the correct Maven/Gradle modules

Depending on your project structure, add these dependencies.

1. VanillaBP adapter dependencies:
   For each BPMS your application uses, add exactly one matching VanillaBP adapter. Add it to the main application module (with several modules) or to the single module which builds the Spring Boot application. The adapter dependency brings the VanillaBP Spring Boot integration along, so you need no other VanillaBP dependency. See Available BPMS adapters for the list of adapters and their documentation.

2. io.vanillabp:vanillabp-spring-boot-support:
   This dependency contains the VanillaBP APIs you need to compile. Add it to pure workflow modules when you have several modules. It keeps workflow modules independent of the concrete BPMS. The APIs of a BPMS are out of reach on purpose, so you cannot break the hexagonal architecture by accident. The module's README lists the APIs it includes. It also brings the integration SPI along. These are the interfaces your application may implement where VanillaBP needs something only your application knows, most of all aggregate persistence.

Aligning versions: the VanillaBP BOM

io.vanillabp:vanillabp-bom manages the versions of all VanillaBP artifacts an application uses. Import it once in the parent POM and leave out the <version> of every VanillaBP dependency:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.vanillabp</groupId>
      <artifactId>vanillabp-bom</artifactId>
      <version>2.0.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

For Gradle use implementation platform("io.vanillabp:vanillabp-bom:2.0.0").

Maven Central lists which patch releases of the 2.0 line exist.

Two things are worth knowing about the BOM:

• BPMS adapters are not part of the BOM. Each adapter is released independently of the platform integration, so its dependency has its own version (see Available BPMS adapters).
• The BOM must never be older than your adapters. A version which your build manages always wins over the version an adapter needs transitively. Maven does this without a word, even if it means a downgrade. The build stays green, but the application would fail at runtime. So adapters check the platform version at startup. If it is too old, they stop with a message which names the artifacts to raise. Raising the BOM to the reported version fixes it.

Configure the transaction outbox

Adapters which connect to a remote BPMS (e.g. a SaaS offering) cannot take part in the local database transaction of your application. This could lead to inconsistencies, for example a workflow started in the BPMS while its workflow aggregate was never stored. To avoid that, VanillaBP starts such workflows in two phases. It plans the second phase through a transaction outbox. That is an entry written in the same local transaction as the workflow aggregate. VanillaBP processes it reliably after the commit, even if the application crashes and restarts in between.

Phase two is a moment, not a kind of call. Phase one is your transaction, and phase two is everything which runs after its commit. A call to a BPMS is one kind of work in phase two. An operation which an extension registered for itself is another. Both are written into the same outbox, in the same transaction, and both run after it committed.

The embedded engine Camunda 7 takes the same path, although it could join your transaction. The reason: an operation which loses a concurrency conflict cannot be repeated inside the caller's transaction. Repeating only the engine part would move the workflow on while your data rolls back. So an application on Camunda 7 needs an outbox as well (see the adapter's wiki).

This works without any setup:

• JPA: entries are stored in the table VANILLABP_PHASE_TWO_OUTBOX. If an entry carries a payload, the payload goes into VANILLABP_PHASE_TWO_OUTBOX_PAYLOAD. The table VANILLABP_HOUSEKEEPING records which node removes the old rows of a store tonight. All three tables are created automatically, and all three belong to io.vanillabp:vanillabp-schema. A Quarkus application runs the same store, written once for both platforms. A payload is deleted when its entry is dispatched. It is deleted again together with the entry when vanillabp.outbox.retention has passed. An entry which still waits, or which is blocked, keeps its payload as long as it is in the table. The housekeeping deletes by age only payloads which no entry names any more. Such payloads are left behind by a write which never committed.
• MongoDB: entries are stored in the collection vanillabp-phase-two-outbox. If an entry carries a payload, the payload goes into vanillabp-phase-two-outbox-payloads. Both collections belong to VanillaBP. MongoDB creates a collection with the first write, so neither has to exist before. vanillabp.outbox.create-schema decides whether VanillaBP creates the indexes for reading them at startup. A payload is deleted when its entry is dispatched. It is deleted again together with the entry when vanillabp.outbox.retention has passed. An entry which still waits, or which is blocked, keeps its payload as long as it is in the collection. The housekeeping deletes by age only payloads which no entry names any more. Such payloads are left behind by a write outside a MongoDB transaction, when that transaction does not commit. A backup should cover both collections, because an entry restored without its bytes cannot be dispatched. For transactional behavior, MongoDB has to run as a replica set, and you have to define a MongoTransactionManager bean. Spring Boot does not configure one for you. Without it the startup ends with a message which names it, because an application whose aggregates no transaction covers cannot start a workflow at all.

The outbox is chosen per workflow aggregate. The entry always goes with the transaction which stores the aggregate. An application may use more than one kind of persistence (e.g. a JPA workflow module and a MongoDB workflow module during a migration). Then both defaults are active at the same time. Each aggregate is served by the outbox which matches its persistence, detected from the aggregate's Spring Data repository. VanillaBP checks at startup that a needed outbox is available. If the adapter with the first priority for a process needs the two-phase start and no outbox can be found, the application does not boot. The message names the ways to fix it.

You can tune the outbox with these properties (defaults shown):

vanillabp:
  outbox:
    # the longest the background poller sleeps while nothing is due. It is a
    # cap and not a rhythm: the poller asks the store when its earliest
    # unfinished entry is due and sleeps until exactly that moment. After a
    # commit the entry is dispatched at once, except while the application is
    # still starting (see "What the stores do differently"). Raise it and an
    # application waiting in a timer stops asking its database at all - see
    # "How late an entry can be dispatched"
    poll-interval: PT10S
    # the distance to the FIRST retry after a failed attempt; every further
    # attempt doubles it until max-attempt-frequency is reached
    attempt-frequency: PT30S
    # the longest distance the growing backoff reaches, so a BPMS which comes
    # back is noticed within it however long it was away
    max-attempt-frequency: PT5M
    # after how many failed attempts an entry is blocked (manual cleanup
    # required); fifty of them span an outage of about four hours
    block-after-attempts: 50
    # how long an entry may wait for a BPMS which does not report its
    # workflow yet, counted from the moment it was written; such a wait uses
    # no attempts. Not set by default, which means the time the attempts above
    # take, about four hours
    # wait-for-visibility-at-most: PT4H
    # how many entries a store dispatches at the same time; the workflow
    # aggregate decides which thread takes an entry - see "Several entries leave
    # at the same time". The JPA store and the MongoDB store both read it
    dispatch-threads: 4
    # how long successfully dispatched entries (marked DONE) are kept before
    # they are deleted asynchronously - see "What the outbox guarantees"
    retention: P7D
    housekeeping:
      # when the outbox removes what it does not need any more: the dispatched
      # entries whose retention passed and the payloads no entry names. Inside
      # the window it works off as much as fits, outside it neither costs
      # anything - see "When the outbox throws things away"
      start: 04:00
      end: 05:00
      # the zone the two times above are read in. Unset means the zone of the
      # JVM, and a JVM standing on UTC without this key is warned once at the
      # startup - it starts either way
      #zone: Europe/Vienna
    # whether to create what the outbox reads automatically - both tables,
    # or both collections on MongoDB, and the indexes on them; disable if
    # the database schema is managed manually (e.g. Flyway/Liquibase). On
    # JDBC both tables are then checked at startup instead
    create-schema: true
    jdbc:
      # disable the JPA default (e.g. if you define your own outbox)
      enabled: true
      # custom table name (the table is created automatically with this name)
      #table: VANILLABP_PHASE_TWO_OUTBOX
      # custom table name for the payloads; unset it follows the table above
      #payload-table: VANILLABP_PHASE_TWO_OUTBOX_PAYLOAD
      # custom name of the table the records of processed task deliveries
      # live in; unset it is VANILLABP_TASK_DELIVERY, and unlike the payloads
      # it does not follow a renamed outbox
      #delivery-table: VANILLABP_TASK_DELIVERY
      # custom name of the table the housekeeping takes its claim in; unset it
      # is VANILLABP_HOUSEKEEPING, and like the deliveries it does not follow a
      # renamed outbox
      #housekeeping-table: VANILLABP_HOUSEKEEPING
    mongo:
      # disable the MongoDB default (e.g. if you define your own outbox)
      enabled: true
      # custom collection name
      collection: vanillabp-phase-two-outbox
      # custom collection name for the payloads; unset it follows the collection above
      #payload-collection: vanillabp-phase-two-outbox-payloads
      # custom name of the collection the records of processed task
      # deliveries live in
      delivery-collection: vanillabp-task-deliveries
      # custom name of the collection the housekeeping takes its claim in
      housekeeping-collection: vanillabp-housekeeping

The store of the payloads is named after the outbox it belongs to. If you set no name for it, VanillaBP adds _PAYLOAD to the table name and -payloads to the collection name. So if you rename the outbox, the payloads are renamed with it. Two applications on one schema need exactly that. They keep apart by the name of the outbox. If the payloads stayed behind, the housekeeping of one application would delete rows of the other.

Each of these names has to be different. The outbox, the payloads behind it and the records of delivered tasks are in the same database. So if two keys name the same table or collection, the startup ends and says which two keys they are. Table names are compared in capitals, the way a relational database looks them up. Collection names are compared as you wrote them.

A custom store also takes on a promise. VanillaBP calls your aggregate persistence during phase two, on whatever thread your store dispatches on. VanillaBP opens the transaction around that call. It is the transaction which belongs to the aggregate the entry names, so loading and saving share it, no matter which store dispatched. Your persistence must not expect the thread which served the request. That thread is long gone by then.

Applications which use another persistence technology (or an outbox infrastructure they already have) can define their own bean which implements io.vanillabp.integration.spi.PhaseTwoOutbox. As soon as there are several outboxes, you have to assign an outbox to specific aggregates. This is also how you move a high-load process to a store of its own. Define a bean which implements io.vanillabp.integration.spi.PhaseTwoOutboxAware<MyAggregate> and returns the outbox to use. The implementation with the most specific aggregate class wins, as for aggregate persistence. Every outbox instance needs its own store (table/collection). Two dispatchers polling the same store would compete and dispatch entries twice.

A custom store stores what the call carries and nothing more: the operation NAME (call.operation()), the workflow module, the BPMN process, the serialized aggregate ID, the optional adapter ID, the arguments and the idempotency key. It never interprets any of them. At dispatch time it hands the stored values back to VanillaBP's router, which works out the operation. Operation names may have a namespace (my-extension:NOTIFY), because extensions add operations of their own. So store the name as text of at least 255 characters.

What the outbox guarantees

• Every planned operation carries an idempotency key (why). The store keeps the key unique AMONG THE ENTRIES STILL WAITING for their dispatch. So if the same operation is planned again before the first one reached the BPMS, nothing happens. The key says that the operation is planned once, not that it ever happened. The same operation after the dispatch is a new one and is carried out. Starting a workflow is keyed by workflow module, BPMN process and aggregate ID. So there is one workflow per aggregate as long as the start is pending. Correlating a message gets a key only if you pass a correlation ID. Without one, the same message may be correlated again, and that is allowed. The key names its operation, so completing and cancelling one task no longer share it. A workflow start by message is the one exception, on purpose: it shares the key of the plain start.
• A younger call may take the place of one still waiting. The rule above drops the second call. That is what a call which names an operation wants. A call which carries a payload wants the opposite. The payload is a state, and the older entry holds the older state. So such a call may ask to replace the entry instead, see work for after the commit. A store of your own which does not support this keeps the older state and writes a warning which names the store. Implement PhaseTwoOutbox.scheduleReplacingWhatIsStillWaiting to support it. Never replace an entry which a dispatch has already taken. (why)
• A dropped plan is logged as a warning. The warning names what was not planned, and both causes which VanillaBP cannot tell apart. Either something which already happened was dispatched again, or a second valid operation lost against one still waiting. See Message correlation for what a loop or a multi-instance activity has to do about it.
• Dispatched entries are marked DONE, not deleted. They are removed in the background after vanillabp.outbox.retention (7 days by default). That keeps them readable for support. It does not make the deduplication window longer, which ends with the dispatch. Make it shorter if the store grows too much, but do not set it to zero. Making it shorter no longer costs anything on the inbound side. The records of delivered tasks have their own vanillabp.delivery.retention (see below). It follows this number only if you do not set it.
• Dispatch is at-least-once. The application may crash after the BPMS accepted an operation, but before the entry is marked DONE. Then the entry is dispatched again after the restart. For workflow starts, VanillaBP first probes the recorded BPMS. If the workflow is already known, it consumes the entry, so a duplicate start is unlikely. The window is small, but not closed, so never build on exactly-once. Your @WorkflowTask methods should be idempotent anyway (see Workflow tasks).
• Entries which keep failing are blocked after block-after-attempts attempts. Then somebody has to look at them, see how long it tries, and how to open a blocked entry again. Watching the outbox store for blocked entries is a job for operations. Alert on vanillabp.outbox.blocked (the metrics). The gauge of waiting entries falls when an entry is blocked, so on its own it would look as if the backlog had cleared.
• A failure which the BPMS answers the same way every time is blocked at once. An adapter may report a failure as permanent (MigratableProcessService.isPhaseTwoFailureRepeatable), for instance a command which Camunda 7 rejects as invalid. Repeating such an operation gains nothing. So the entry is blocked after the first attempt and logged as an error. The error names the operation, the workflow module, the BPMN process and the aggregate. Everything else stays repeatable, and that is why losing a concurrency conflict does no lasting harm. (why)

When the outbox throws things away

Two kinds of row leave the store: the entries which were dispatched longer ago than vanillabp.outbox.retention, and the payloads which no entry names any more. Both used to be removed at the end of every poll, so every application paid for them all day long. Now this happens in a window at night, from vanillabp.outbox.housekeeping.start to .end. That is four to five in the morning unless you set other times. Inside the window the store does as much as fits. Outside the window, removing rows costs nothing. (why)

You do not set how much fits. The first batch of a night is a thousand rows. VanillaBP measures how long it took. The next batch is twice as large, as long as twice that time still fits in the rest of the window. A batch which runs past the end halves the size. The next night starts at half of the largest batch which fitted, because your database will not be the same at four in the morning tomorrow.

Only one node does the housekeeping of a store per night. If two nodes did it at once, each one would measure the other's work. So a node writes a claim into the store's own database, in the table VANILLABP_HOUSEKEEPING or the collection vanillabp-housekeeping. The other nodes leave that store alone until the window closes. The claim is per STORE. So in an application which runs a JPA outbox and a MongoDB outbox side by side, two different nodes may do the housekeeping.

A node which is down for the whole window does no housekeeping that night. A pod scaled to zero at night, or one which is being rolled out at that moment, simply skips it. VanillaBP has no mechanism to catch up, because what such a mechanism should do would be a guess. The next night takes the backlog with it, and the three meters below show it happen.

Three meters say whether the window is wide enough: vanillabp.outbox.housekeeping.remaining is what the window did not get to, .removed is how many rows it removed, and .window.used is how much of the window it needed. All three are set when the window closes, and they stay until the next one closes. Read them together. If the window was used up AND left something behind, the window is too small, and you should make it wider. The remainder alone would not tell you that. A store with work left may also be one which never did its housekeeping, because the node was down.

The window is read in a time zone, and an application on UTC is told so once. A container runs on UTC unless somebody sets its zone. "Four in the morning" then means four UTC, which in most places is the middle of the working day. Such an application starts, and VanillaBP warns while it starts. The line names the hours at which the window really runs and the zone of the JVM. It also names two ways to choose another zone, and neither needs a new build. Set the zone of the container (TZ=Europe/Vienna), or set the environment variable of the property (VANILLABP_OUTBOX_HOUSEKEEPING_ZONE=Europe/Vienna). Write UTC there if you mean UTC, and the line goes away. It stays a warning because UTC is the normal case for a container. A refused start would cost you a deployment. A window in the wrong zone only costs a cleanup at an unexpected hour.

What the stores do differently

Every store keeps the guarantees above. How each one keeps them differs:

Store	How a duplicate is recognised	What happens when the same key is scheduled again	How a re-dispatch is recognised
JPA (the default)	a unique DEDUP_KEY column. It holds the key while the entry waits, and the entry's own id once it is DONE	while it waits, the second schedule is dropped and logged. After the dispatch the key is free, so the same operation is a new one	STATUS, ATTEMPTS and NEXT_ATTEMPT_AT of the row, plus LEASED_BY. An attempt which ended is counted there, and one whose node died left its name in the claim
MongoDB (Spring Boot)	a unique index on the entry's dedup field. It holds the key while the entry waits, and the entry's own id once it is DONE	while it waits, the second schedule is dropped and logged. After the dispatch the key is free, so the same operation is a new one	the status, the attempt count and the claim of the document, the same way
JDBC (Quarkus)	the store of the first row, running the same code	same	same
MongoDB (Quarkus)	as on Spring Boot	same	same

All stores wait for VanillaBP in the same way. Spring Boot answers requests before the BPMN models have reached the BPMS. So the application may be asked to start a workflow whose process the BPMS does not have yet. An entry planned in that window waits for the first poll and does not go out at once. This happens once per application start. It costs the time until the poller starts, not a full vanillabp.outbox.poll-interval. If the entry were sent right away, the answer would be "no such process". That is one of the failures which are blocked on the first attempt, and a start which was only early would wait for somebody to repair it.

The store which used to be here

While 2.0 was being built, a Spring Boot application with JPA kept its phase-two entries in the gruelbox transaction outbox, in the table TXNO_OUTBOX. A version 1 application never wrote that table (why). Now VanillaBP writes its own store, and the gruelbox store lives in its own repository. An application which wants to keep it adds the artifact io.vanillabp:gruelbox-phase-two-outbox and changes no configuration. That repository documents what the store does differently and what it cannot answer.

An application may have run such a snapshot and then upgrade without that artifact. At startup it is told how many entries are left in TXNO_OUTBOX, because nothing reads that table any more. Dispatch them with the snapshot you came from, and drop the table afterwards. The artifact is what asks for that store, and the key vanillabp.outbox.gruelbox.enabled belongs to it. With the artifact, false hands the entries back to VanillaBP's own store. Without the artifact, the line does nothing at all.

Several entries leave at the same time

The outbox dispatches up to vanillabp.outbox.dispatch-threads entries at once, four unless you set another number. A free thread does not just take the next entry. The workflow aggregate of the entry decides which thread takes it. So two operations of one workflow reach the BPMS in the order in which they were planned, while operations of different workflows run at the same time. Giving the next entry to the next free thread would have fixed the throughput but broken the order. Nothing later in the chain would have noticed, until a customer did. A broadcast signal names no aggregate. So it is ordered by its BPMN process instead, and two signals of one process keep their order as well. (why)

Raise the number if the BPMS is so slow that the threads wait for it and not for the database. Every dispatch costs a database connection and a call to the BPMS. So a bigger number only moves the limit into the connection pool, where it is harder to see. In a thread dump, the dispatching threads are the daemon threads named vanillabp-outbox-dispatch-0 and up. Next to them is the poller vanillabp-outbox, which feeds them.

This does not keep the order of an entry whose dispatch FAILED. That entry waits for its backoff, and the next entry of the same aggregate passes it in the meantime. A single thread did the same, because a failed entry goes back into the table either way.

The MongoDB store dispatches the same way and reads the same setting.

How late an entry can be dispatched

The poll interval used to answer this question, but it does not any more. The poller asks its store when the earliest entry it still owes something to is due, and it sleeps until exactly that moment. So an entry goes out at its due time, whatever poll-interval says. After a commit, the entry goes out at once, as it always did. The interval limits only one case: work which a node wrote down before it went away. The node which writes an entry dispatches it itself, so nothing tells the other nodes about that row. The limit is what makes somebody look anyway (see running more than one node).

So the number controls cost, not latency. Left at ten seconds, an application keeps exactly the timing it always had. Set to ten minutes, an application with nothing to do pays one small query per store every ten minutes, instead of 8640 polls a day. The only thing it gives up is how soon the work of a node which died is picked up. Setting it back to seconds because an entry looked late buys nothing. That entry was waiting for its own due time, not for the poller.

When you change it, the startup log says what else is still awake. Read it before you conclude that the database has gone quiet. The retention cleanup of the task delivery records still looks once an hour, and only after a delivery was recorded. The gauge which counts the entries waiting in an outbox asks for them while your monitoring scrapes, at most once per vanillabp.metrics.gauge-cache. And how an adapter finds work for your handlers is up to that adapter, and its own documentation explains it.

How long it tries, and how to open a blocked entry again

A failed dispatch is retried at a growing distance: attempt-frequency for the first retry, doubled for every attempt after it, up to max-attempt-frequency. With the defaults that is thirty seconds, then one minute, two, four, and five minutes from there on. That makes fifty attempts in total, which is close to four hours of trying. The numbers answer two questions at once. The limit says how long after a BPMS comes back the first entry reaches it: five minutes at worst. The attempts say how long an outage the store survives without anybody's help. Four hours is a cluster upgrade, and that is the length they were chosen for. If your BPMS will be down longer than that, raise block-after-attempts before the maintenance, not after it.

When a node has claimed an entry, that node holds it, and the hold lasts longer than the distance above. A poll claims an entry by writing down which node took it and how long the claim lasts: one attempt-frequency. From the claim on, the node keeps moving that moment forward until the dispatch is over. This starts at the claim and not at the dispatch, because an entry first waits for the thread its aggregate travels on. A wait longer than the claim would hand the entry to the next poll. So an operation which queues for its thread and then needs a minute against a slow BPMS is carried out once, not once per attempt-frequency. No second node picks it up in the meantime. If the node which carries it disappears, nothing moves the moment forward any more. The claim runs out, and the next poll of any node takes over the entry. That is the recovery which was always there, now with its own deadline instead of one tied to the retry distance.

block-after-attempts counts attempts, not claims. An attempt is counted when it ENDED, with whatever result it had. So an entry which travels for a long time uses nothing of the budget, and an entry which keeps failing uses one per failure. A slow BPMS is therefore never the reason why an entry is blocked. On JPA the columns are LEASED_BY and LEASED_UNTIL, on MongoDB the fields leasedBy and leasedUntil. Both are empty as long as nobody holds the entry.

Two things are worth knowing about how the distance grows. It applies to a dispatch which FAILED. An entry whose workflow the BPMS has not made searchable yet is a different case. It gets its own due time, which is the adapter's window and not this backoff. And every store which VanillaBP owns computes the growth itself.

Waiting for a read model is not an attempt. Sometimes the BPMS does not report the workflow of an entry yet, for example a Camunda 8 cluster whose exporter is behind. Such an entry uses none of block-after-attempts. It comes back after the adapter's window as often as needed. It is blocked only when wait-for-visibility-at-most has passed since it was written. That key is not set by default. Then it is the time which the attempts take with the growing backoff, about four hours. So a read model which stopped gets as much time as a BPMS which is away. Counting each answer as an attempt used to block entries after fifty windows, which were eight minutes of a stopped exporter. And those entries did not come back when the exporter caught up. An entry which was blocked this way says so in its ERROR, with the time it waited.

Every store uses the window exactly as the adapter named it. It is not read as an upper limit. A window longer than attempt-frequency delays the entry, and a shorter one brings it forward. The adapter knows when its BPMS can answer, and asking earlier only asks in vain. So the same adapter answers the same way, whichever store you chose. That is why an adapter's documentation can name a number at all.

A blocked entry stays in the store, and no retention removes it. The operation still has to happen, and an entry deleted before it is finished is a workflow which silently never goes on. But a blocked entry no longer stands in the way of its own repetition. On the stores VanillaBP owns, blocking releases the deduplication key, exactly as a dispatched entry releases it. So the application can plan that operation again, and the new attempt is an entry of its own. Whoever repairs the blocked row has to expect a second row for the same operation. The counter vanillabp.outbox.discarded no longer rises in this case, because there is nothing left to drop.

A blocked entry says why. On the stores VanillaBP owns, the column LAST_FAILURE (on MongoDB the field lastFailure) holds the reason of the last attempt which did not get through. That is the class and the message of what the dispatch threw, and of its causes, in one line and cut to 1000 bytes. The stack trace stays in the log. Every attempt writes the reason, and an attempt which gets through empties it. So an entry which is still being repeated says why as well. The gruelbox store keeps no reason. Blocked outbox entries has the query which reads it.

The payload of a blocked entry stays with it. The retention counts from the entry. It deletes a dispatched entry together with the bytes which that entry named. It never takes the bytes of an entry which is still waiting or blocked. So an entry which is opened again weeks later still carries the state its caller planned it with. The cleanup by age deletes only payloads which no entry names any more. Such payloads are left behind by a write which never committed.

Keeping those bytes costs something while the entry stays blocked. Before the housekeeping removes a payload by age, it asks whether an entry still names it. It asks in the database: one statement per batch, and nothing of either set travels through your application. On MongoDB the entries are joined with a $lookup, using the index on the reference inside the entry document. On a relational database the outbox table has a column PAYLOAD_REFERENCE with an index on it, so the question is a lookup there too. We measured it on PostgreSQL 16.15 on 2026-09-25. With a hundred blocked entries, one pass took 1 ms, no matter whether ten thousand or a million dispatched entries were next to them. With ten thousand blocked entries it took 20 ms. Repairing the blocked entries makes the cost go away, and you should do that anyway.

How to find a blocked entry, open it again or delete it, on JPA and on MongoDB, is on its own page: Blocked outbox entries. The ERROR which reports a blocked entry links there too. That page also says what to check before you open an entry, because by then the operation may have a younger entry.

What VanillaBP remembers about delivered tasks

This is the inbound side of the outbox. A remote BPMS may hand out the same task again. VanillaBP records what it processed. So it answers a repeated delivery from that record and does not run the handler twice (see Workflow tasks). The store is chosen per workflow aggregate, exactly like the outbox. So a record always goes with the aggregate's own transaction:

• JPA/JDBC aggregates get a table VANILLABP_TASK_DELIVERY. It is written through a connection bound to the transaction which Spring manages. If your database has naming rules, vanillabp.outbox.jdbc.delivery-table gives the table a name of your own, the way table and payload-table do it for the outbox. It is a separate table, not the outbox table, on purpose. The outbox holds calls which wait to be dispatched. A delivery record is a fact to be read back. And the same store answers which BPMS holds a task when you complete it. That is what its TASK_ID column and the index on it are for.
• MongoDB aggregates get a collection vanillabp-task-deliveries, keyed by the delivery. So the document ID makes each record unique. If your database has naming rules, vanillabp.outbox.mongo.delivery-collection gives the collection a name of your own, the way collection and payload-collection do it for the outbox. Records go with the MongoDB transaction which Spring manages. That needs a replica set and a MongoTransactionManager. Without one, a record is written at once and removed again if its transaction does not commit. So a repeated delivery of work which was rolled back is not skipped. Only a process which dies in between leaves a record behind.
• The STORE settings are those of the outbox, but the retention is not: vanillabp.outbox.create-schema decides whether VanillaBP creates the table or the index itself. vanillabp.outbox.jdbc.enabled / .mongo.enabled switch the defaults off. vanillabp.outbox.mongo.delivery-collection names the MongoDB collection. And vanillabp.delivery.retention decides how long a record is kept after the BPMS last asked about it. That last one is a property of its own, because it decides correctness, not how long support can read along. A delivery which arrives later than the retention finds no record and runs your handler a second time. On the outbox side, the deduplication window ends with the dispatch, and the retention only keeps a dispatched entry readable. If you do not set it, it follows vanillabp.outbox.retention, where the number lived before the two were separated. VanillaBP says at startup which of the two applies. It is read for the whole application, not per workflow module, because one cleanup per store deletes by age across the whole table. The BPMS asks about a task your application leaves open until somebody completes it. So the record of that task stays, while the retention deletes the records of the tasks which are done.
• For any other persistence, write a bean of your own which implements io.vanillabp.integration.spi.TaskDeliveryLog. If there are several, assign them to aggregates through io.vanillabp.integration.spi.TaskDeliveryLogAware<MyAggregate>, exactly as for the outbox. Without any store, the startup says that deliveries are not deduplicated and names the way out. Such a store decides for itself how long a record lives. If the record of an open task has to survive in the same way, implement stillOpen. VanillaBP calls it with the key of every repeated delivery it answers.
• vanillabp.delivery.release-on-workflow-end (default false, can be changed per workflow module as vanillabp.workflow-modules.<id>.delivery.release-on-workflow-end) lets a workflow which ended delete its own records, instead of waiting for the retention, see Workflow tasks. Both stores above implement the deletion. If a store of your own does not, the startup names it together with the property.

Creating the tables with Liquibase or Flyway

In some applications the schema is a reviewed artifact which the deployment pipeline applies. Such an application switches VanillaBP's table creation off and applies the statements itself:

vanillabp:
  outbox:
    create-schema: false

With that switch off, VanillaBP checks at startup that the tables in your relational database exist. If one is missing, it ends the boot and names the table, the property and the artifact to apply. That is better than failing at the first workflow start, hours after the deployment. (why)

The statements come from the artifact io.vanillabp:vanillabp-schema. It contains nothing but schema definitions. So a repository which holds your migrations can depend on it without pulling in the runtime:

<dependency>
  <groupId>io.vanillabp</groupId>
  <artifactId>vanillabp-schema</artifactId>
</dependency>

With Liquibase, include the changelog from the classpath in your own:

<include file="vanillabp/schema/changelog.xml" />

A table name you configured is a changelog property. So a renamed table needs no copy of the changelog. Set vanillabp.outbox.table, vanillabp.delivery.table, vanillabp.payload.table or vanillabp.housekeeping.table as a changelog parameter. Write the same name on both sides, or the runtime looks for a table which the changelog did not build. Do not forget the payload table when you rename the outbox. The runtime names it after the outbox table. So a changelog which renames only vanillabp.outbox.table builds a payload table which VanillaBP will not look for. The delivery table and the housekeeping table are not derived from anything. They follow only vanillabp.outbox.jdbc.delivery-table or .housekeeping-table.

The generated Flyway files cannot follow a name of yours. They are generated while io.vanillabp:vanillabp-schema is built. Nobody knows your names then, so they carry the defaults. If you rename a table, apply the changelog with Liquibase, or edit the generated statements before you apply them.

The changelog also carries the indexes, and they are not decoration. Each time the poller wakes up, it asks its store when the earliest entry it owes something to is due. Without an index, that question reads the whole table. We measured it on PostgreSQL 16 with 200000 rows: 11 to 18 ms without the index against 0.05 ms with it. The first number grows with the table, the second does not. The outbox table gets three indexes. Each one filters the same status and then reads a moment of its own: when the next entry is due, what the retention may delete, and how long the oldest waiting entry has already waited. The last one answers the gauge vanillabp.outbox.oldest.pending.age. We measured it on PostgreSQL 16.15 in September 2026. Without the index it took 0.17 ms with a backlog of 1000 entries, 28 ms with 100000 and 78 ms with 500000. With the index it took 0.07 ms at every size. The table which holds the payloads has its own index, on the moment a payload was written. The delivery table gets one for what its retention deletes by, plus the ones its reads need. The fourth table holds the claim of the nightly housekeeping and has no index. It has one row per store, and every read of it reads by that row's key. An application which applied an earlier version of the changelog gets the indexes from the changeset of the version which added them. If VanillaBP created the tables of an application, the startup says which index is missing and which statement adds it. Creating an index on a large table locks it, so that is your decision and not something a boot should do behind your back.

With Flyway, apply the generated SQL (vanillabp/schema/flyway/<database>/V<version>__vanillabp_schema.sql in the same artifact) with a second Flyway instance of its own: its own locations and its own history table. That matters. Flyway keeps one version timeline per history table. So VanillaBP's versions would sooner or later clash with the numbers of your own migrations. With flyway_schema_history_vanillabp as the history table, both timelines stay separate, and you can upgrade each side without touching the other.

Every instance which is not the first one needs two settings, and both are required:

Flyway
    .configure()
    .dataSource(dataSource)
    .locations("classpath:vanillabp/schema/flyway/postgresql")
    .table("flyway_schema_history_vanillabp")
    .baselineOnMigrate(true)   // the schema already holds the tables of the other owners
    .baselineVersion("0")      // the default baseline is 1, which would skip V1.0.0 silently
    .load();

baselineOnMigrate lets an instance work in a schema which is not empty, and next to another history that is always the case. Without it, Flyway finds tables it has no history for and refuses with Found non-empty schema(s) "PUBLIC" but no schema history table. baselineVersion is needed as well. The default baseline is version 1, so every migration numbered 1.0.0 counts as applied and is skipped. Flyway then reports success, but the table never appears.

A FlywayMigrationInitializer per instance makes sure the schema exists before anything reads it. Spring Boot's Flyway module lets the entity manager factory depend on beans of that type, not on Flyway beans. So an instance without an initializer migrates whenever its bean happens to be created, and that may be after Hibernate looked.

Which databases: tests cover H2 and PostgreSQL. MySQL, MariaDB, SQL Server, Oracle and DB2 ship as well, generated by Liquibase from the same changelog, but nothing here runs them. It is the same list the Camunda 7 engine serves, so VanillaBP never supports fewer databases than the engine. Oracle and DB2 count a VARCHAR in bytes. What that means for the ID of your aggregate is in Workflow aggregates.

The blueprints persistence-liquibase and persistence-flyway build and test both ways. They include the three owners of schema which a workflow application usually has: VanillaBP, the workflow module and the application itself.

The same switch on MongoDB names indexes, not tables

vanillabp.outbox.create-schema reaches the MongoDB stores as well, but there it means something else. MongoDB creates a collection with its first document. So there is no collection to create, and none to check either. What VanillaBP would have created are the indexes which answer its questions. With the switch off, they are your job:

db.getCollection("vanillabp-phase-two-outbox").createIndex({ "dedupKey": 1 }, { "unique": true });
db.getCollection("vanillabp-phase-two-outbox").createIndex({ "status": 1, "nextAttemptAt": 1 });
db.getCollection("vanillabp-phase-two-outbox").createIndex({ "status": 1, "doneAt": 1 });
db.getCollection("vanillabp-phase-two-outbox").createIndex({ "status": 1, "createdAt": 1 });
db.getCollection("vanillabp-phase-two-outbox").createIndex({ "args.payloadReference": 1 }, { "sparse": true });
db.getCollection("vanillabp-phase-two-outbox-payloads").createIndex({ "createdAt": 1 });
db.getCollection("vanillabp-task-deliveries").createIndex({ "lastSeenAt": 1 });
db.getCollection("vanillabp-task-deliveries").createIndex({ "taskId": 1 });
db.getCollection("vanillabp-task-deliveries").createIndex({ "aggregateId": 1 });
db.getCollection("vanillabp-task-deliveries").createIndex({ "workflowId": 1 });

These are the default names. If you gave a collection a name of your own with vanillabp.outbox.mongo.collection, vanillabp.outbox.mongo.payload-collection or vanillabp.outbox.mongo.delivery-collection, use that name instead.

You do not have to copy this list to find out what is missing. At startup, VanillaBP reads which indexes each of the three collections has. It warns about every missing index, with the statement above and a word on what reads it. That stays a warning. The application boots, and until the index exists, the question reads the whole collection. That costs more the longer the application runs. Act on the unique index on dedupKey first. It is not about speed. Without it, two nodes which write the same operation at the same moment both plan it, and the BPMS is called twice.

The Camunda 7 engine tables are the engine's job too. Its JAR ships a Liquibase changelog (org/camunda/bpm/engine/db/liquibase/camunda-changelog.xml) and plain create scripts. Refer to those instead of copying them, because the engine version decides what is correct.

Configure the adapters

Each BPMS the application uses is configured as a named adapter below vanillabp.adapters. The key is the adapter ID, which names one instance of a BPMS. Its type names the BPMS adapter which serves it.

Zero configuration: the classpath IS the configuration

An application with one BPMS adapter dependency and one workflow module needs no vanillabp.* property at all. VanillaBP works out everything it can by itself:

• the adapter: the single adapter type found in the classpath becomes the one configured adapter (its ID is the adapter type). Because it is the only one, it is also the prioritized adapter;
• the workflow module: every module found in the classpath (a META-INF/workflow-module descriptor) is known without a section of its own;
• the BPMN location: see the resources-location convention.

You only have to write the settings which the BPMS itself needs (e.g. the address of a remote cluster). They go into the adapter's own section:

vanillabp:
  adapters:
    camunda8:                               # the ID names the adapter type ...
      rest-address: http://localhost:8080   # ... only the BPMS' own settings remain

The conventions at work:

• The adapter's type defaults to its ID. You need vanillabp.adapters.<id>.type only for a custom adapter ID (old-bpms, c7-legacy, …) which does not match an adapter type. Custom IDs let you run several instances of the same BPMS. VanillaBP can never derive them, so they always need their type. This default exists for applications which upgrade from VanillaBP 1, where there was one adapter and its ID was its type. It is not a recommendation. A good ID says WHICH system it is (camunda8-saas next to camunda8-onprem, camunda7-loans next to camunda7-claims). An operator reads it in a log line, in a metric tag and in a process definition id. And a migration from one Camunda 8 configuration to another needs two names anyway, because both cannot be called camunda8. An adapter ID must never be renamed while workflows are running. VanillaBP stores it to know what belongs to which BPMS. See naming adapter ids, and never renaming them.
• A single configured adapter is the prioritized adapter. You may leave out vanillabp.prioritized-adapters as long as exactly one adapter is configured.
• vanillabp.retired-adapters names the adapter ids which this application USED to have. At startup, VanillaBP asks its stores for which ids they still hold waiting outbox entries or records of open tasks. It names any such id which is no longer configured. Either the id was renamed, which loses workflows, or it was removed too early. Naming it here says that you know about the leftovers, see naming adapter ids, and never renaming them.

Several BPMS: naming the order is the whole configuration

With several adapters in the classpath, VanillaBP never guesses which one starts new workflows. But as long as the adapter IDs are adapter types, naming the order is all you need:

vanillabp:
  prioritized-adapters:
    - camunda8    # new workflows start here
    - camunda7    # existing workflows complete here

VanillaBP derives the adapter sections from that list. You have to write down everything which differs from the convention. A custom ID needs its type. Two IDs of the same type must be distinguishable. What that means is up to the adapter, e.g. separate databases or separate cluster endpoints.

vanillabp.prioritized-adapters defines the order in which adapters are used. You can set again which adapter is asked first for a workflow module and for a single workflow: the levels a setting is written at. New workflows always start in the first adapter of the list which counts for them. Existing workflows are found by asking the adapters in order of priority (cached after the first successful election). That is what makes BPMS migration possible without touching business code.

vanillabp:
  prioritized-adapters:
    - camunda8    # new workflows start here
    - legacy      # existing workflows complete here
  adapters:
    camunda8:
      rest-address: http://localhost:8080   # ID = type, so 'type' can be omitted
    legacy:
      type: camunda7    # a custom ID needs its type
      # optional: do not abort booting if deployment to this (non-first-priority)
      # BPMS fails, e.g. because the old BPMS is temporarily unreachable during
      # a migration; 'fail' (the default) aborts booting
      deployment-failure: warn

Settings of one adapter (e.g. the connection to a remote BPMS) go into the SAME section. Each adapter adds its own keys to vanillabp.adapters.<id>.* (see your adapter's wiki for its keys). There is no separate namespace per adapter. A key which differs by scope may be written again for a workflow module, for a workflow and for a single task: the levels a setting is written at.

deployment-failure: warn only works for adapters which are not first in the priority list, neither for the workflow module nor for any of its workflows. If an adapter has the first priority anywhere, a failed deployment to it always stops the boot. Otherwise new workflows could not be started.

Migrating a single workflow (workflow-level overrides)

You can override the prioritized adapters for a single workflow (BPMN process) of a workflow module. This is how you move one process to a new BPMS while the rest of the module stays:

vanillabp:
  prioritized-adapters:
    - camunda7
    - camunda8
  workflow-modules:
    loan-approval:
      prioritized-adapters:
        - camunda7    # the module still runs on Camunda 7...
        - camunda8
      workflows:
        RiskAssessment:
          prioritized-adapters:
            - camunda8    # ...but new 'RiskAssessment' workflows start in Camunda 8
            - camunda7

Two things happen automatically:

• Deployment union: the resources of the workflow module are deployed to every adapter named at the module level or at any workflow level. Each such adapter gets all the resources of the module, because deployment works per file while the election works per process. So an adapter named only for a single workflow still gets the deployment it needs to start that workflow.
• Validation: a workflow ID below workflows may match no executable BPMN process of the module. That does not stop the boot, because the BPMN may arrive later during a migration. But a warning at startup reports it and names the known BPMN process IDs. If a prioritized-adapters entry at workflow level names an unknown adapter ID, the boot stops, as on the other levels.

Overriding configuration via environment variables

You can override all vanillabp.* properties with environment variables (e.g. VANILLABP_ADAPTERS_CAMUNDA8_DEPLOYMENT_FAILURE=warn). One limit applies on all platforms. An environment variable can only override an adapter or workflow module which a configuration file already declares. It cannot introduce a new entry whose ID contains dashes or dots. The binder cannot rebuild the ID from the variable's name (C8_CLOUD could mean c8-cloud, c8.cloud or a nested key). VanillaBP does not silently ignore such a variable. It fails the startup with a message which names the variable, the configured IDs and the fix: declare the ID in a configuration file (e.g. vanillabp.adapters.<id>.type) and use environment variables to override its values.

Look twice at a key with a dash in it, because the name of a variable is not the name of the property. Spring Boot writes the property name in capitals, turns every dot into an underscore and drops every dash. So vanillabp.adapters.camunda8.rest-address becomes VANILLABP_ADAPTERS_CAMUNDA8_RESTADDRESS. That form works wherever the key stands. Spring Boot also understands a dash written as an underscore, as long as every dash of the name is written that way. So VANILLABP_ADAPTERS_CAMUNDA8_REST_ADDRESS arrives too. This stops working where the part with the dash is a section with an ID below it. Spring Boot has to read that ID out of the variable, and then WORKFLOW_MODULES reads as workflow.modules. So VANILLABP_WORKFLOW_MODULES_LOAN_RESOURCES_LOCATION reaches nothing, but VANILLABP_WORKFLOWMODULES_LOAN_RESOURCESLOCATION does. Write the form without dashes, and it is right in both places.

Quarkus reads the other spelling. There every dash is always an underscore, and the name without it reaches nothing (see the same section on the Quarkus page). So an application deployed on both platforms, or a pipeline which runs one example on both, sets two variables for one property. Nothing reports the wrong one. A variable which matches no property is not read, and the setting keeps the value from the configuration file.