Blocked outbox entries

The phase-two outbox is the table (or MongoDB collection) which holds work planned for after your commit, for example a call to the BPMS (business process management system). It gives up on an entry in three cases, and from then on the entry waits for a person. This page is for that person. It shows how to find such an entry, how to open it again or delete it, and what that does to the deduplication of the operation. Every ERROR which reports a blocked entry links here.

1. Why an entry is blocked
   The three causes, and the two ERROR lines one of them can leave behind.
2. Finding a blocked entry
   One query for the table, one for the MongoDB collection, both with the reason.
3. Opening it again
   The statement which makes the entry due again, and the one case where it needs a second column.
4. Deleting it
   When that is the right answer, and what happens to its payload.
5. What happens to the idempotency key
   Why the operation may now have two entries, and what to check before you open the older one.
6. The gruelbox store
   Where the steps for TXNO_OUTBOX are.

Everything here is a statement against your database. A user interface for it is planned. Until it exists, a database client is the tool.

Why an entry is blocked

An entry is blocked when

• the adapter or an extension says that repeating cannot help. Such an entry had exactly one attempt. The Business Cockpit, for example, gives up on a change whose workflow or user task still does not exist after ten minutes.
• vanillabp.outbox.block-after-attempts attempts failed. With the defaults this takes about four hours, see how long it tries.
• the entry waited longer than vanillabp.outbox.wait-for-visibility-at-most for a BPMS which still did not report the workflow.

Nothing dispatches a blocked entry again, and no retention deletes it. The ERROR of the outbox names the operation, the workflow and the id of the entry, and it ends with a link to this page. An extension which gave up may write an ERROR of its own next to it. That one says what to check in the extension's terms. The outbox line says which row it is and has the stack trace. Read both.

The entry also keeps the reason, so you can still read it after the log was rotated. The column is LAST_FAILURE, on MongoDB the field lastFailure. It holds the class and the message of what the dispatch threw, and of its causes, in one line. The stack trace is not in it. The text is cut to 1000 bytes, and a cut text ends with .... For an entry which waited too long for its BPMS, the text starts with how long it waited and ends with the last answer of the adapter.

Every attempt writes this field, not only the one which blocks the entry. An attempt which failed, or which the BPMS answered with "not yet", writes its reason. An attempt which got through empties the field. So an entry which is still being repeated also tells you why.

vanillabp.outbox.blocked counts blocked entries (the metrics). It is the number to alert on, because vanillabp.outbox.pending falls when an entry is blocked.

Finding a blocked entry

On a relational database (JPA on Spring Boot, JDBC on Quarkus):

SELECT ID, OPERATION, WORKFLOW_MODULE_ID, BPMN_PROCESS_ID, AGGREGATE_ID, ADAPTER_ID,
       IDEMPOTENCY_KEY, ATTEMPTS, CREATED_AT, LAST_FAILURE
  FROM VANILLABP_PHASE_TWO_OUTBOX
 WHERE STATUS = 'BLOCKED'
 ORDER BY CREATED_AT;

On MongoDB:

db.getCollection('vanillabp-phase-two-outbox')
  .find({ status: 'BLOCKED' })
  .sort({ createdAt: 1 })

ID (_id on MongoDB) is the id the ERROR names. LAST_FAILURE (lastFailure) is why the entry is blocked. The names above are the defaults. Where you set vanillabp.outbox.jdbc.table or vanillabp.outbox.mongo.collection, use your name.

Opening it again

Fix the cause first. Then make the entry open and due again. On a relational database:

UPDATE VANILLABP_PHASE_TWO_OUTBOX
   SET STATUS = 'OPEN', ATTEMPTS = 0, LEASED_BY = NULL, LEASED_UNTIL = NULL,
       NEXT_ATTEMPT_AT = CURRENT_TIMESTAMP
 WHERE ID = '...' AND STATUS = 'BLOCKED';

On MongoDB:

db.getCollection('vanillabp-phase-two-outbox').updateOne(
    { _id: '...', status: 'BLOCKED' },
    { $set: { status: 'OPEN', attempts: 0, nextAttemptAt: new Date() },
      $unset: { leasedBy: '', leasedUntil: '' } })

ATTEMPTS goes back to zero, because the dispatcher only reads entries whose attempts are below block-after-attempts. The claim is cleared, because no poll takes an entry which somebody holds. A blocked entry usually has no claim. But it does have one if its node died while sending the entry. Nothing tells the poller about your statement, so the entry goes out at its next wake-up. That is at most vanillabp.outbox.poll-interval away. If you restart a node, it picks the entry up at once.

The statements leave LAST_FAILURE (lastFailure) as it is. The next attempt writes it: a new reason if it fails again, nothing if it gets through.

An entry blocked because it waited too long for its BPMS needs one more column. The wait is counted from CREATED_AT (createdAt), and the statement above does not change it. If the BPMS still answers "not yet", the entry is blocked again at that first answer. Open it once the BPMS reports the workflow. Or also set CREATED_AT = CURRENT_TIMESTAMP (createdAt: new Date()), which gives it the whole wait again. The entry then also goes behind younger entries of the same workflow, because entries of one workflow leave in the order of that column.

When you open an entry, at-least-once becomes real. An entry may be blocked because the BPMS answered too late, not because it did not answer at all. Then the reopened entry sends the operation there a second time. A workflow start checks the BPMS first, and does nothing if the workflow is already there. Completing a task which is already completed does nothing and writes a warning. A message correlation can really do its work twice. So read an entry which carries one before you open it.

Deleting it

DELETE FROM VANILLABP_PHASE_TWO_OUTBOX WHERE ID = '...' AND STATUS = 'BLOCKED';

db.getCollection('vanillabp-phase-two-outbox').deleteOne({ _id: '...', status: 'BLOCKED' })

Then the operation is gone, and a workflow which waited for it does not go on. Delete an entry if nobody wants the operation any more, or if a younger entry carries it (see the next section).

A payload which the entry named stays in VANILLABP_PHASE_TWO_OUTBOX_PAYLOAD (vanillabp-phase-two-outbox-payloads) for a while. You do not have to do anything about it. The housekeeping removes a payload which no entry names once vanillabp.outbox.retention passed.

What happens to the idempotency key

The idempotency key says that an operation is planned only once. While an entry waits, a second plan of the same operation is discarded. Blocking an entry frees its key, in the same way as sending it does: DEDUP_KEY (dedupKey) then holds the entry's own id. IDEMPOTENCY_KEY (idempotencyKey) keeps the key so you can still read it. So when the application plans the same operation after the block, the new plan is not discarded. It becomes an entry of its own, and the blocked one stays.

Opening the blocked entry does not take the key back. It deduplicates nothing from then on. Check for a younger entry of the same operation before you open it, with the IDEMPOTENCY_KEY the query above returned:

SELECT ID, STATUS, CREATED_AT, DONE_AT
  FROM VANILLABP_PHASE_TWO_OUTBOX
 WHERE IDEMPOTENCY_KEY = '...' AND ID <> '...';

db.getCollection('vanillabp-phase-two-outbox').find({ idempotencyKey: '...', _id: { $ne: '...' } })

• Nothing found: nobody planned the operation again. Opening the blocked entry is the only way it happens.
• An entry which is OPEN or DONE: the operation was planned again, and it has gone out or will go out. Opening the blocked one sends it a second time, so deleting it is usually the better answer.

"Nothing found" has one limit. A DONE entry is deleted once vanillabp.outbox.retention passed, seven days by default. A blocked entry older than that may have had a younger twin which is already gone.

The gruelbox store

An application which runs the store of io.vanillabp:gruelbox-phase-two-outbox keeps its entries in gruelbox' table TXNO_OUTBOX, with gruelbox' columns. The steps follow the same idea, but differ in detail. That store frees the key of a blocked entry when the same operation is planned again, not at the moment of the block. It keeps no reason in the row either, so there the log is the only place which says why. Its README has the statements: a blocked entry and the way back.