Possible Race condition on canceling a workflow instance

[Zelldon]

Describe the bug

Reported by a cloud user was that it seem that operate get out of sync. But there real problem was that the workflow instance get stuck during canceling the instance.

Imagine the following workflow:

Task B is completed and during completing and taking the next sequence flow the workflow instance is canceled by the user. What now can happen is that the canceling will not clean up correctly all scopes and the instance get stuck. In our case only the Task A and the Sub Process was terminated correctly. The multi instance and workflow instance was still alive.

To Reproduce

This is also reproducible via an engine unit test and the following process
multiBug.bpmn.txt.

Test

/*
 * Copyright © 2020  camunda services GmbH (info@camunda.com)
 *
 *  Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *        http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 *
 */
package io.zeebe.engine.processor.workflow.multiinstance;
import io.zeebe.engine.util.EngineRule;

import io.zeebe.engine.util.RecordToWrite;

import io.zeebe.model.bpmn.Bpmn;

import io.zeebe.protocol.record.intent.JobIntent;

import io.zeebe.protocol.record.intent.WorkflowInstanceIntent;

import io.zeebe.protocol.record.value.BpmnElementType;

import io.zeebe.test.util.record.RecordingExporter;

import io.zeebe.test.util.record.RecordingExporterTestWatcher;

import java.util.Arrays;

import java.util.stream.Collectors;

import org.junit.ClassRule;

import org.junit.Rule;

import org.junit.Test;
public class MultiInstanceBugTest {
@ClassRule public static final EngineRule ENGINE = EngineRule.singlePartition();

public static final String TASK_ELEMENT_ID = "task";

private static final String PROCESS_ID = "process";

private static final String SUB_PROCESS_ELEMENT_ID = "sub-process";

private static final String JOB_TYPE = "test";

private static final String INPUT_COLLECTION = "items";

private static final String INPUT_ELEMENT = "item";
@Rule

public final RecordingExporterTestWatcher recordingExporterTestWatcher =

new RecordingExporterTestWatcher();
@Test

public void shouldActivateStartEventForEachElement() {

// given

final var resourceAsStream =

MultiInstanceBugTest.class.getResourceAsStream("/workflows/multiBug.bpmn");

final var bpmnModelInstance = Bpmn.readModelFromStream(resourceAsStream);

ENGINE.deployment().withXmlResource(bpmnModelInstance).deploy();
final long workflowInstanceKey =
    ENGINE
        .workflowInstance()
        .ofBpmnProcessId(PROCESS_ID)
        .withVariable(INPUT_COLLECTION, Arrays.asList(10, 20, 30))
        .create();

final var instanceRecordValueRecord =
    RecordingExporter.workflowInstanceRecords()
        .withIntent(WorkflowInstanceIntent.ELEMENT_ACTIVATED)
        .withElementType(BpmnElementType.PROCESS)
        .getFirst();

final var taskA =
    RecordingExporter.jobRecords().withIntent(JobIntent.CREATED).withType("a").getFirst();
final var taskB =
    RecordingExporter.jobRecords().withIntent(JobIntent.CREATED).withType("b").getFirst();
ENGINE.writeRecords(RecordToWrite.command().job(JobIntent.COMPLETE).key(taskB.getKey()));

RecordingExporter.jobRecords().withIntent(JobIntent.COMPLETED).withType("b").getFirst();
//    ENGINE.stop();

// when
ENGINE.writeRecords(
    RecordToWrite.command()
        .key(workflowInstanceKey)
        .workflowInstance(WorkflowInstanceIntent.CANCEL, instanceRecordValueRecord.getValue()));

//    ENGINE.start();

// then
final var instanceCanceled =
    RecordingExporter.workflowInstanceRecords()
        .withIntent(WorkflowInstanceIntent.ELEMENT_TERMINATED)
        .withElementType(BpmnElementType.PROCESS)
        .getFirst();

final var collect =
    RecordingExporter.workflowInstanceRecords()
        .withIntent(WorkflowInstanceIntent.ELEMENT_TERMINATED)
        .collect(Collectors.toList());

}

}

Be aware that this is a race condition, which means that the test might not fail on the first try.

Expected behavior

The workflow instance can be terminated without any problems.

Log/Stacktrace
We have extracted the records from the failed scenario you can find them in the
records.txt

We see that only the task and sub process are terminated and the sequence flow after the task B is taken. Actually the same sequence flow seems to be taken twice, but it has different scope ids, maybe related to the problem. Be aware that we can't share here the actual bpmn process to protect our user. So this means the output above doesn't match to our current model which we have shown in the example.

BUT if we run the test we can see similar output
output-test.txt

Environment:

• Zeebe Version: 0.23.0

[menski]

Support Case: https://jira.camunda.com/browse/SUPPORT-7623

Waiting for customer to prioritize