-
Notifications
You must be signed in to change notification settings - Fork 603
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reorganize stream processor and engine tests #10458
Conversation
bors r+ |
10443: Do not take a backup if it already exists r=deepthidevaki a=deepthidevaki ## Description After restore, the log is truncated to the checkpoint position. So the checkpoint record is processed again and will trigger a new backup with the same Id of the backup it restored from. With this PR, `BackupService` handles this case gracefully. In addition, we also do not take a new backup if existing backup is failed or in progress. Alternatively, we can delete this backup and take a new one. But chances of it happening (i.e triggering a new backup when one already is in progress/failed) is very low. So we can keep this simple. ## Related issues closes #10430 10450: fix(raft): handle exceptions on partition server init r=megglos a=megglos ## Description Previously any RuntimeException happening in RaftPartitionServer#initServer lead to a broken future chain during start which lead to a stale node without any logs on the actual exception occurred during init. Ultimately flying silently till [here](https://github.com/camunda/zeebe/blob/main/broker/src/main/java/io/camunda/zeebe/broker/bootstrap/PartitionManagerStep.java#L42) bringing the startup to a halt. With this change issues are transparent, see this [log](https://console.cloud.google.com/logs/query;cursorTimestamp=2022-09-22T11:20:44.904454673Z;query=resource.labels.namespace_name%3D%22medic-cw-37-de38e9e086-benchmark-mixed%22%0Aresource.labels.pod_name%3D%22medic-cw-37-de38e9e086-benchmark-mixed-zeebe-2%22%0A-resource.labels.container_name%3D%22debugger-9q4tw%22%0A-logName%3D%22projects%2Fzeebe-io%2Flogs%2Fevents%22%0Atimestamp%3D%222022-09-22T11:20:44.904454673Z%22%0AinsertId%3D%2238ntsbk0c2ikn344%22%0Atimestamp%3D%222022-09-22T11:20:44.904454673Z%22%0AinsertId%3D%2238ntsbk0c2ikn344%22;summaryFields=:false:32:beginning;timeRange=2022-09-22T10:20:44.905Z%2F2022-09-22T11:20:44.905Z?project=zeebe-io) from a pod created with this change. This bug was hiding the underlying issue a node not being able to start due to #10451 . ## Related issues relates to #10451 10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: Meggle (Sebastian Bathke) <sebastian.bathke@camunda.com> Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed (retrying...): |
10450: fix(raft): handle exceptions on partition server init r=megglos a=megglos ## Description Previously any RuntimeException happening in RaftPartitionServer#initServer lead to a broken future chain during start which lead to a stale node without any logs on the actual exception occurred during init. Ultimately flying silently till [here](https://github.com/camunda/zeebe/blob/main/broker/src/main/java/io/camunda/zeebe/broker/bootstrap/PartitionManagerStep.java#L42) bringing the startup to a halt. With this change issues are transparent, see this [log](https://console.cloud.google.com/logs/query;cursorTimestamp=2022-09-22T11:20:44.904454673Z;query=resource.labels.namespace_name%3D%22medic-cw-37-de38e9e086-benchmark-mixed%22%0Aresource.labels.pod_name%3D%22medic-cw-37-de38e9e086-benchmark-mixed-zeebe-2%22%0A-resource.labels.container_name%3D%22debugger-9q4tw%22%0A-logName%3D%22projects%2Fzeebe-io%2Flogs%2Fevents%22%0Atimestamp%3D%222022-09-22T11:20:44.904454673Z%22%0AinsertId%3D%2238ntsbk0c2ikn344%22%0Atimestamp%3D%222022-09-22T11:20:44.904454673Z%22%0AinsertId%3D%2238ntsbk0c2ikn344%22;summaryFields=:false:32:beginning;timeRange=2022-09-22T10:20:44.905Z%2F2022-09-22T11:20:44.905Z?project=zeebe-io) from a pod created with this change. This bug was hiding the underlying issue a node not being able to start due to #10451 . ## Related issues relates to #10451 10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Meggle (Sebastian Bathke) <sebastian.bathke@camunda.com> Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed (retrying...): |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed: |
bors retry |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed: |
bors retry |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 10482: deps(maven): bump snakeyaml from 1.32 to 1.33 r=Zelldon a=dependabot[bot] Bumps [snakeyaml](https://bitbucket.org/snakeyaml/snakeyaml) from 1.32 to 1.33. <details> <summary>Commits</summary> <ul> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/eafb23ec31a0babe591c00e1b50e557a5e3f9a1d"><code>eafb23e</code></a> [maven-release-plugin] prepare for next development iteration</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/26624702fab8e0a1c301d7fad723c048528f75c3"><code>2662470</code></a> Improve JavaDoc</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/80827798f06aeb3d4f2632b94075ca7633418829"><code>8082779</code></a> Always emit numberish strings with quotes</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/42d6c79430431fe9033d3ba50f6a7dc6798ba7ad"><code>42d6c79</code></a> Reformat test</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/1962a437263348c3b90857cda4bbfa2bd97908f8"><code>1962a43</code></a> Refactor: rename variables in Emitter</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/bc594ad6e2b87c3fc26844e407276796fd866a40"><code>bc594ad</code></a> Issue 553: honor code point limit in loadAll</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/c3e98fd755a949f65cf11f2ff39e55a1c2afd1c2"><code>c3e98fd</code></a> Update changes.xml</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/a06f76859f2f07580b1d9fa6b66ea84aaad26cf8"><code>a06f768</code></a> Remove deprecated Tag manipulation</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/5a0027a3781b92f59bf92cdeb1b7590589993efd"><code>5a0027a</code></a> Remove unused WhitespaceToken</li> <li><a href="https://bitbucket.org/snakeyaml/snakeyaml/commits/3f05838828b8df36ab961bf836f373b8c20cb8ff"><code>3f05838</code></a> Improve JavaDoc</li> <li>Additional commits viewable in <a href="https://bitbucket.org/snakeyaml/snakeyaml/branches/compare/snakeyaml-1.33..snakeyaml-1.32">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.yaml:snakeyaml&package-manager=maven&previous-version=1.32&new-version=1.33)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: Christopher Zell <zelldon91@googlemail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Build failed (retrying...): |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed: |
bors r+ |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed: |
Weird |
bors r+ |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
Build failed: |
@oleschoenburg I added a fix which I also have observed in #10526 (comment) I have seen when running Please have a another look at the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Zelldon
I think there's something wrong now 😅:
Error: Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.375 s <<< FAILURE! - in io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceTest
Error: io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceTest.shouldPreserveOrderingOfWritesEvenWithRetries Time elapsed: 2.639 s <<< FAILURE!
org.mockito.exceptions.verification.VerificationInOrderFailure:
Verification in order failure:
logStreamBatchWriter.tryWrite();
Wanted 1 time:
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceTest.shouldPreserveOrderingOfWritesEvenWithRetries(ProcessingScheduleServiceTest.java:275)
But was 5001 times:
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceImpl.lambda$toRunnable$5(ProcessingScheduleServiceImpl.java:165)
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceImpl.lambda$toRunnable$5(ProcessingScheduleServiceImpl.java:165)
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceImpl.lambda$toRunnable$5(ProcessingScheduleServiceImpl.java:165)
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceImpl.lambda$toRunnable$5(ProcessingScheduleServiceImpl.java:165)
-> at io.camunda.zeebe.streamprocessor.ProcessingScheduleServiceImpl.lambda$toRunnable$5(ProcessingScheduleServiceImpl.java:165)
.....
👍 for splitting the fix in a seperate PR.
Yeah I saw that sorry 🙈 I'm on it. Will create a separate PR. 👍 |
10527: deps(maven): bump software.amazon.awssdk:bom from 2.17.282 to 2.17.283 r=oleschoenburg a=dependabot[bot] Bumps [software.amazon.awssdk:bom](https://github.com/aws/aws-sdk-java-v2) from 2.17.282 to 2.17.283. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md">software.amazon.awssdk:bom's changelog</a>.</em></p> <blockquote> <h1><strong>2.17.283</strong> <strong>2022-09-27</strong></h1> <h2><strong>AWS Cost Explorer Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release is to support retroactive Cost Categories. The new field will enable you to retroactively apply new and existing cost category rules to previous months.</li> </ul> </li> </ul> <h2><strong>AWS SDK for Java v2</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>Updated service endpoint metadata.</li> </ul> </li> </ul> <h2><strong>AWSKendraFrontendService</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>My AWS Service (placeholder) - Amazon Kendra now provides a data source connector for DropBox. For more information, see <a href="https://docs.aws.amazon.com/kendra/latest/dg/data-source-dropbox.html">https://docs.aws.amazon.com/kendra/latest/dg/data-source-dropbox.html</a></li> </ul> </li> </ul> <h2><strong>Amazon Location Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds place IDs, which are unique identifiers of places, along with a new GetPlace operation, which can be used with place IDs to find a place again later. UnitNumber and UnitType are also added as new properties of places.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/8af53203a1e879237b2362d3e26b86949a159d98"><code>8af5320</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/aws/aws-sdk-java-v2/issues/2154">#2154</a> from aws/staging/531bec79-3529-4b44-b7a1-f6feb1ded95e</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/d3bc7121fd663478fdaf764aad5a9848b33730b3"><code>d3bc712</code></a> Release 2.17.283. Updated CHANGELOG.md, README.md and all pom.xml.</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/e069e7bc3f8563dcd10c119eb26aac26ea769fe1"><code>e069e7b</code></a> Updated endpoints.json.</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/63a9497c648f39e6dded6a38422dd225ed03d075"><code>63a9497</code></a> Amazon Location Service Update: This release adds place IDs, which are unique...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/ee3958dd062c97be648d64ca49877bb3935c0b5d"><code>ee3958d</code></a> AWS Cost Explorer Service Update: This release is to support retroactive Cost...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/0abace4f454b92914f76bde97ff7de95ba665aae"><code>0abace4</code></a> AWSKendraFrontendService Update: My AWS Service (placeholder) - Amazon Kendra...</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/0939324e88ff893f012bbb94715d3e1d90da9617"><code>0939324</code></a> Make WebIdentityTokenFileCredentialsProvider implement AutoClosable (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-java-v2/issues/3440">#3440</a>)</li> <li><a href="https://github.com/aws/aws-sdk-java-v2/commit/57677db7068b707f241b8cd26b1261e8987869a6"><code>57677db</code></a> Update to next snapshot version: 2.17.283-SNAPSHOT</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-java-v2/compare/2.17.282...2.17.283">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=software.amazon.awssdk:bom&package-manager=maven&previous-version=2.17.282&new-version=2.17.283)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> 10557: Do not copy ResultBatch everytime r=Zelldon a=Zelldon ## Description I have seen when running shouldPreserveOrderingOfWritesEvenWithRetries that it sometimes can happen that the writing (retry loop) can take up to 3 seconds, where the timeout is 2 seconds. I think this is related to the that we always copy again in the loop (which is not necessary). I also have observed in #10526 (comment) Related to #10458 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10458 related to #10526 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
this test is more related to the state
this test was related to the engine state (processing + replay)
563c420
to
5375eac
Compare
@oleschoenburg thanks for reviewing the other :) I rebased this one, could you accept again? |
bors r+ |
10458: Reorganize stream processor and engine tests r=Zelldon a=Zelldon ## Description Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 related to #10130 10556: fix(helm): rename podSecurityContext to containerSecurityContext r=Zelldon a=megglos ## Description The recent v8.0.14 release introduced containerSecurityContext, see camunda/camunda-platform-helm#374 and was actually failing now with the format `podSecurityContext` was set. ``` Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(StatefulSet.spec.template.spec.securityContext): unknown field "capabilities" in io.k8s.api.core.v1.PodSecurityContext ``` //cc `@aabouzaid` Co-authored-by: Christopher Zell <zelldon91@googlemail.com> Co-authored-by: Meggle (Sebastian Bathke) <sebastian.bathke@camunda.com>
Build failed (retrying...): |
Build succeeded: |
10468: Migrate StreamProcessor test r=Zelldon a=Zelldon ## Description Migrate the StreamProcessor to the new StreamProcessorExtension and Junit5. I completely iterated over all tests and rewrote them. This is part of #10455 which is necessary to split up into StreamPlatform into a separate module (#10130 ) Blocked by #10458 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> related to #10455 Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
/backport |
\backport |
Successfully created backport PR #10585 for |
Description
Moved some tests around to make it easier to detect which need to be migrated for #10455 and to make it easier to create the new module and copy the tests, which are part of the StreamProcessor see #10130
Related issues
related to #10455
related to #10130
Definition of Done
Not all items need to be done depending on the issue and the pull request.
Code changes:
backport stable/1.3
) to the PR, in case that fails you need to create backports manually.Testing:
Documentation:
Please refer to our review guidelines.