Batch processing might drop responses #11848
Assignees
Labels
area/observability
Marks an issue as observability related
area/reliability
Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected)
component/stream-platform
kind/bug
Categorizes an issue or PR as a bug
severity/high
Marks a bug as having a noticeable impact on the user with no known workaround
target:8.2
Issue must be completed before this target release
version:8.1.10
Marks an issue as being completely or in parts released in 8.1.10
version:8.2.0
Marks an issue as being completely or in parts released in 8.2.0
Comments
korthout
added
kind/bug
This is the part that Zeebe Process Automation can do. |
|
7 tasks
|
The problem is a bit wider even. CreateProcessInstanceWithResult can throw away any response that led to the completion of a process instance. If the process ends with a job, the job worker completing this job will not receive a response acknowledging the completion. |
|
@oleschoenburg and me will look at it tomorrow. |
14 tasks
ghost
pushed a commit
to camunda/zeebe-process-test
that referenced
this issue
Feb 28, 2023
677: Sync with Zeebe SNAPSHOT r=koevskinikola a=korthout ## Description <!-- Please explain the changes you made here. --> There were several issues stacked that stopped zeebe-process-test from passing the CI against the latest Zeebe SNAPSHOT version. I've resolved these: - Add while true starting at key to InMemoryDbColumnFamily - Add delete resource RPC gateway implementation - Add logger for `COMMAND_DISTRIBUTION` records - Disable a flaky test due to bug camunda/camunda#11848 I suggest to review this by commit, as these go into more details. ## Related issues <!-- Which issues are closed by this PR or are related --> closes #673 <!-- Cut-off marker _All lines under and including the cut-off marker will be removed from the merge commit message_ ## Definition of Ready Please check the items that apply, before requesting a review. You can find more details about these items in our wiki page about [Pull Requests and Code Reviews](https://github.com/camunda/zeebe/wiki/Pull-Requests-and-Code-Reviews). * [ ] I've reviewed my own code * [ ] I've written a clear changelist description * [ ] I've narrowly scoped my changes * [ ] I've separated structural from behavioural changes --> ## Definition of Done <!-- Please check the items that apply, before merging or (if possible) before requesting a review. --> _Not all items need to be done depending on the issue and the pull request._ Code changes: * [x] The changes are backwards compatibility with previous versions * [ ] If it fixes a bug then PRs are created to backport the fix Testing: * [x] There are unit/integration tests that verify all acceptance criterias of the issue * [ ] New tests are written to ensure backwards compatibility with further versions * [ ] The behavior is tested manually Documentation: * [ ] Javadoc has been written * [ ] The documentation is updated Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
Zelldon
changed the title
ghost
closed this as
14 tasks
ghost
pushed a commit
that referenced
this issue
Mar 1, 2023
11867: [Backport 8.1] fix: never drop responses during batch processing r=oleschoenburg a=Zelldon ## Description Backported #11865, had to fix several merge conflict related to structural changes which have been done on main, but not on 8.1 Had to fix another test, because this was removed on main, but still exists on 8.1 [test: adjust test case](aae1a80) <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> closes #11848 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com> Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
korthout
added a commit
that referenced
this issue
Mar 10, 2023
These two cases failed recently due to a bug introduced with batch processing [1]. The situation happened because multiple commands were being processed and responded in the same batch, while the batch processing assumed only a single response would be writen. As this affected behavior visible to users, it makes sense to introduce qa integration tests that show case that the CreateProcessInstanceWithResult can actually respond when the process instance is completed by another user command: - complete job - publish mesage See: [1] #11848
14 tasks
ghost
pushed a commit
that referenced
this issue
Mar 20, 2023
11993: Add QA tests that verify CreateProcessInstanceWithResult responses r=korthout a=korthout ## Description <!-- Please explain the changes you made here. --> These two cases failed recently due to a bug introduced with batch processing [1]. As this affected behavior visible to users, it makes sense to introduce QA integration tests that showcase that the `CreateProcessInstanceWithResult` can actually respond when the process instance is completed by another user command. ## Related issues <!-- Which issues are closed by this PR or are related --> relates to #11848 Co-authored-by: Nico Korthout <nico.korthout@camunda.com>
backport-action
pushed a commit
that referenced
this issue
Mar 20, 2023
These two cases failed recently due to a bug introduced with batch processing [1]. The situation happened because multiple commands were being processed and responded in the same batch, while the batch processing assumed only a single response would be writen. As this affected behavior visible to users, it makes sense to introduce qa integration tests that show case that the CreateProcessInstanceWithResult can actually respond when the process instance is completed by another user command: - complete job - publish mesage See: [1] #11848 (cherry picked from commit b490894)
backport-action
pushed a commit
that referenced
this issue
Mar 20, 2023
These two cases failed recently due to a bug introduced with batch processing [1]. The situation happened because multiple commands were being processed and responded in the same batch, while the batch processing assumed only a single response would be writen. As this affected behavior visible to users, it makes sense to introduce qa integration tests that show case that the CreateProcessInstanceWithResult can actually respond when the process instance is completed by another user command: - complete job - publish mesage See: [1] #11848 (cherry picked from commit b490894)
ghost
pushed a commit
that referenced
this issue
Mar 20, 2023
ghost
pushed a commit
that referenced
this issue
Mar 20, 2023
ghost
pushed a commit
that referenced
this issue
Mar 21, 2023
ghost
pushed a commit
that referenced
this issue
Mar 21, 2023
npepinpe
added
version:8.2.0
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I encountered this bug when a test in zeebe-process-test became flaky.
The test creates a process instance and waits for the result. The process instance can only be completed by publishing a message. Depending on which arrives at the engine first, the publish message rpc receives a response.
After talking to @Zelldon and @oleschoenburg about this, we believe this problem arises because of batch processing. The implementation assumes that only a single response will be written by the Engine in a batch of processed commands. This assumption is invalid.
To Reproduce
Expected behavior
🏗️ With batch processing enabled, a call to publish a message should be responded to, even during the processing of a call to create process instance with result. I.e., batch processing should allow sending multiple responses (at max one per processed command).
🔧 We should have a test for this case in Zeebe, not just in zeebe-process-test.
Log/Stacktrace
Full Stacktrace
Environment:
8.2.0-alpha5)The text was updated successfully, but these errors were encountered: