IndexOutOfBounds if output collection of multi-instance is modified

[pihme]

Describe the bug

I deployed a BPMN process with a parallel multi-instance embedded subprocess. The multi-instance subprocess defines an output collection to collect the results of the iterations. If the output collection is modified during the iteration (e.g. the size of the output collection is reduced) then multi-instance can't be completed. The processing fails with an IndexOutOfBounds exception. The concrete exception may change depending on the modification.

To Reproduce

test-multi-instance-output-collection.bpmn.txt

1. deploy a BPMN process with a parallel multi-instance embedded subprocess

• define an output collection variable results

2. create an instance of the process
3. modify the variable results when the multi-instance is active (e.g. set the variable to [] - an empty array)

• set variables via command
• set variables via job worker
• set variable via output mapping (e.g. on the start event of the embedded subprocess)

4. verify that the multi-instance is not completed
5. verify that a related error record is written

Expected behavior

An incident is created if the output element can't be added to the output collection. The incident is visible to the user and allows to fix the issue manually by modifying the output collection.

Log/Stacktrace

Full Stacktrace

java.lang.IndexOutOfBoundsException: index=8 capacity=8
	at org.agrona.concurrent.UnsafeBuffer.boundsCheck(UnsafeBuffer.java:2212) ~[agrona-1.15.0.jar:1.15.0]
	at org.agrona.concurrent.UnsafeBuffer.getByte(UnsafeBuffer.java:947) ~[agrona-1.15.0.jar:1.15.0]
	at io.camunda.zeebe.msgpack.spec.MsgPackReader.skipValues(MsgPackReader.java:355) ~[zeebe-msgpack-core-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.msgpack.spec.MsgPackReader.skipValue(MsgPackReader.java:350) ~[zeebe-msgpack-core-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.insertAt(MultiInstanceBodyProcessor.java:410) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.lambda$updateOutputCollection$17(MultiInstanceBodyProcessor.java:389) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.Either$Right.map(Either.java:355) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.updateOutputCollection(MultiInstanceBodyProcessor.java:380) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.lambda$updateOutputCollection$16(MultiInstanceBodyProcessor.java:366) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at java.util.Optional.map(Unknown Source) ~[?:?]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.updateOutputCollection(MultiInstanceBodyProcessor.java:364) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.beforeExecutionPathCompleted(MultiInstanceBodyProcessor.java:153) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.MultiInstanceBodyProcessor.beforeExecutionPathCompleted(MultiInstanceBodyProcessor.java:35) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.behavior.BpmnStateTransitionBehavior.lambda$beforeExecutionPathCompleted$5(BpmnStateTransitionBehavior.java:354) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.behavior.BpmnStateTransitionBehavior.invokeElementContainerIfPresent(BpmnStateTransitionBehavior.java:442) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.behavior.BpmnStateTransitionBehavior.beforeExecutionPathCompleted(BpmnStateTransitionBehavior.java:350) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.behavior.BpmnStateTransitionBehavior.transitionToCompleted(BpmnStateTransitionBehavior.java:164) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.SubProcessProcessor.lambda$onComplete$3(SubProcessProcessor.java:76) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.Either$Right.flatMap(Either.java:366) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.SubProcessProcessor.onComplete(SubProcessProcessor.java:73) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.container.SubProcessProcessor.onComplete(SubProcessProcessor.java:23) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.BpmnStreamProcessor.processEvent(BpmnStreamProcessor.java:133) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.BpmnStreamProcessor.lambda$processRecord$0(BpmnStreamProcessor.java:110) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.Either$Right.ifRightOrLeft(Either.java:381) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.bpmn.BpmnStreamProcessor.processRecord(BpmnStreamProcessor.java:107) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:54) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.lambda$processInTransaction$3(ProcessingStateMachine.java:300) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:84) ~[zeebe-db-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:290) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.processCommand(ProcessingStateMachine.java:253) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.tryToReadNextRecord(ProcessingStateMachine.java:213) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.engine.processing.streamprocessor.ProcessingStateMachine.readNextRecord(ProcessingStateMachine.java:189) ~[zeebe-workflow-engine-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) ~[zeebe-util-8.0.0.jar:8.0.0]
	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) ~[zeebe-util-8.0.0.jar:8.0.0]

Environment:

• Zeebe Version: 8.0.0 (and others)

[npepinpe]

We'll mark it for next release, ideally before - the result here is that the instance is blacklisted (all instances for this process presumably) which is still not observable for users, and looking at the logs doesn't give you much of an idea of what to fix either. There is no known workaround at the moment, but this might be because we still haven't root caused the issue.