Flaky MultiPartitionDeploymentLifecycleTest.shouldTestLifecycle

[lenaschoenburg]

Summary

Try to answer the following as best as possible

• How often does the test fail? More than once
• Does it block your work? No
• Do we suspect that it is a real failure? No

Failures

Outline known failure cases, e.g. a failed assertion and its stacktrace obtained from Jenkins

Example assertion failure

Error:  io.camunda.zeebe.engine.processing.deployment.MultiPartitionDeploymentLifecycleTest.shouldTestLifecycle  Time elapsed: 0.601 s  <<< FAILURE!
java.lang.AssertionError: 
Actual and expected have the same elements but not in the same order, at index 6 actual element was:

(COMPLETED, EVENT)

whereas expected element was:

(COMPLETE, COMMAND)
at io.camunda.zeebe.engine.processing.deployment.MultiPartitionDeploymentLifecycleTest.shouldTestLifecycle(MultiPartitionDeploymentLifecycleTest.java:59)

Hypotheses

The ordering of events from multiple deployment distribution attempts is not guaranteed, the tests seems to be too strict.

[remcowesterhoud]

Happened again https://github.com/camunda/zeebe/runs/7666727813?check_suite_focus=true

[pihme]

Increased severity as this week this has been the main reason why builds failed to flaky tests

[Zelldon]

Occurred again here https://github.com/camunda/zeebe/actions/runs/3150907128/jobs/5124227135

[Zelldon]

Happened again on main:

Warning:  Flakes: 
Warning:  io.camunda.zeebe.engine.processing.deployment.MultiPartitionDeploymentLifecycleTest.shouldTestLifecycle
Error:    Run 1: MultiPartitionDeploymentLifecycleTest.shouldTestLifecycle:73 
Expecting actual:
  [(CREATE, COMMAND, 1),
    (COMPLETE, COMMAND, 3),
    (COMPLETE, COMMAND, 2),
    (CREATED, EVENT, 1),
    (CREATED, EVENT, 1),
    (DISTRIBUTING, EVENT, 2),
    (DISTRIBUTING, EVENT, 3),
    (COMPLETED, EVENT, 3),
    (COMPLETED, EVENT, 2),
    (FULLY_DISTRIBUTED, EVENT, 1)]
to start with:
  [(CREATE, COMMAND, 1), (CREATED, EVENT, 1), (CREATED, EVENT, 1)]

https://github.com/camunda/zeebe/actions/runs/3225048569/jobs/5276879600

[deepthidevaki]

This is failing quite frequently now.

[remcowesterhoud]

I was giving some priority to the hacktoberfest contributions, but I will start digging into this again today.

So far my observation is that it fails because in the exporter we get all the commands first for some reason.

[remcowesterhoud]

After running it a a lot of times I finally got a failure locally. This allowed me to see the positions of the exported records. What is odd about this is the 3rd line in this log. Here we can see that the DeploymentDistribution.COMPLETE command has position 2.

Position: 1 TypedRecordImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=CREATE, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}
Position: 1 TypedRecordImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=DISTRIBUTE, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}
Position: 2 TypedRecordImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=COMPLETE, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":2}}
Position: 1 TypedRecordImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=DISTRIBUTE, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}
Position: 3 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=CREATED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=PROCESS, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"bpmnProcessId":"shouldReDistributeAfterRecovery","version":1,"processDefinitionKey":2251799813685249,"resourceName":"process.bpmn","checksum":"XkCnzocF31uHoqDJOltpSQ==","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgI...}
Position: 4 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=CREATED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}
Position: 5 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=DISTRIBUTING, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":2}}
Position: 6 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=DISTRIBUTING, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":3}}
Position: 7 TypedRecordImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=COMPLETE, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":3}}
Position: 8 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=COMPLETED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":2}}
Position: 9 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=COMPLETED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT_DISTRIBUTION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"partitionId":3}}
Position: 10 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=FULLY_DISTRIBUTED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[],"processesMetadata":[],"decisionRequirementsMetadata":[],"decisionsMetadata":[]}}
Position: 2 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=DISTRIBUTED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}
Position: 2 TypedRecordImpl{metadata=RecordMetadata{recordType=EVENT, intentValue=255, intent=DISTRIBUTED, requestStreamId=-2147483648, requestId=-1, protocolVersion=3, valueType=DEPLOYMENT, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.2.0}, value={"resources":[{"resourceName":"process.bpmn","resource":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxkZWZpbml0aW9ucyB4bWxuczpicG1uZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvREkiIHhtbG5zOmRjPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9ERC8yMDEwMDUyNC9EQyIgeG1sbnM6ZGk9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0RELzIwMTAwNTI0L0RJIiBpZD0iZGVmaW5pdGlvbnNfZmNkYTJlY2MtYzBhYS00NDNmLWI5NDItNWZkNjFkMmQzMjVkIiB0YXJnZXROYW1lc3BhY2U9Imh0dHA6Ly93d3cub21nLm9yZy9zcGVjL0JQTU4vMjAxMDA1MjQvTU9ERUwiIHhtbG5zPSJodHRwOi8vd3d3Lm9tZy5vcmcvc3BlYy9CUE1OLzIwMTAwNTI0L01PREVMIj4KICA8cHJvY2VzcyBpZD0ic2hvdWxkUmVEaXN0cmlidXRlQWZ0ZXJSZWNvdmVyeSIgaXNFeGVjdXRhYmxlPSJ0cnVlIj4KICAgIDxzdGFydEV2ZW50IGlkPSJzdGFydEV2ZW50XzgxY2FhNmQ2LTA5YmUtNDgzZC05YjZkLTg4NGZlODYyYjU2YiI+CiAgICAgIDxvdXRnb2luZz5zZXF1ZW5jZUZsb3dfODJmMjYzOTAtOWM4MS00NzhiLTgzM2YtYmQwZmViZmVlNmFiPC9vdXRnb2luZz4KICAgIDwvc3RhcnRFdmVudD4KICAgIDxlbmRFdmVudCBpZD0iZW5kRXZlbnRfM2NiODUyZDgtMThiYS00YjBlLTg2MGMtOTg1ZDc2ODZhYWMwIj4KICAgICAgPGluY29taW5nPnNlcXVlbmNlRmxvd184MmY...}

Looking at the code we can see that we always write a DeploymentDistribution.DISTRIBUTING event before we start distributing to other partitions:

    otherPartitions.forEach(
        partitionId -> {
          deploymentDistributionRecord.setPartition(partitionId);
          stateWriter.appendFollowUpEvent(
              key, DeploymentDistributionIntent.DISTRIBUTING, deploymentDistributionRecord);

          distributeDeploymentToPartition(key, partitionId, copiedDeploymentBuffer);
        });

Somehow we are writing the response of this deployment distribution to the log stream before we are writing the distributing event.

Hypothesis
I am not sure how to verify this, but I feel like this could be a result from the engine abstraction. My theory is that it has to do with the buffering of records and writing until the command has been processed fully.

1. We receive a Deployment.CREATE command
2. During processing we will write the DeploymentDistribution.DISTRIBUTING event. During processing we also send a message to the other partitions in order to distribute the deployment.
3. The DeploymentDistribution.DISTRIBUTING event is written to the buffer. Nothing is written to the log stream yet. The other partition receives the message and writes a Deployment.DISTRIBUTE command. At this point this partition is idle so it will immediately start processing this command.
4. Here we run into a race condition. If the second partition sends the response back to the first partition before the first partition has finished processing the Deployment.CREATE command it will write the DeploymentDistribution.COMPLETE before it writes the buffered events.

Tl;dr: I think the inter partition communication does not take event buffering into consideration. Resulting in a strange order of commands/events on the log stream.

@Zelldon Could you verify if my hypothesis makes any sense?

[Zelldon]

Sounds plausible, but I think this was already before the abstraction like that, since the LogStreamWriters also just "buffered" the records during processing. I guess there was just some timing change which might cause this issue more often.

[remcowesterhoud]

🤔 There was also the changes in how the deployment distribution works somewhat recently. It might be that this caused a change in the timing. I can't fully remember the details of this change.

I think the order shouldn't matter either way so I will modify the test to be a bit more lenient in the command ordering and that should fix the flakiness.

[Zelldon]

@remcowesterhoud maybe it makes sense to use the PostCommitTasks for the Deployment distribution? So the request would only be executed IF the processing is done and the transaction is committed. Might make more sense I guess?