ZeebePartition caught in a loop transition to inactive after a dead partition #9924
Assignees
Labels
kind/bug
Categorizes an issue or PR as a bug
severity/low
Marks a bug as having little to no noticeable impact for the user
version:8.1.3
Marks an issue as being completely or in parts released in 8.1.3
version:8.2.0-alpha1
Marks an issue as being completely or in parts released in 8.2.0-alpha1
version:8.2.0
Marks an issue as being completely or in parts released in 8.2.0
Comments
deepthidevaki
added
kind/bug
|
After an initial investigation, it looks like this is caused due to transitions initiated by raft due to cluster configuration operations. When we force raft to transition to inactive, it is not committed to the configuration. So the other nodes still have marked it as active. This can cause it to transition back to active, when it receives a new update from the leader. |
|
Actually, not able to reproduce this exact behavior in 8.2.0-SNAPSHOT. After inserting a deliberate ZeebeDbInconistentException in streamprocessor, what I observe is the following: StreamProcessor in Leader fails with the unrecoverable exception -> detected as dead Expected behavior is once it is inactive because the partition is dead, it doesn't come back to active. |
10 tasks
zeebe-bors-camunda bot
added a commit
that referenced
this issue
Oct 24, 2022
10759: deps(maven): bump version.zpt from 8.1.1 to 8.1.2 r=oleschoenburg a=dependabot[bot] Bumps `version.zpt` from 8.1.1 to 8.1.2. Updates `zeebe-process-test-assertions` from 8.1.1 to 8.1.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/camunda/zeebe-process-test/releases">zeebe-process-test-assertions's releases</a>.</em></p> <blockquote> <h2>8.1.2</h2> <h2>What's Changed</h2> <ul> <li>Zeebe release 8.1.2 by <a href="https://github.com/korthout"><code>`@korthout</code></a>` in <a href="https://github-redirect.dependabot.com/camunda/zeebe-process-test/pull/545">camunda/zeebe-process-test#545</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2">https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/camunda/zeebe-process-test/commit/30bf84155a40f169f99918beb8743d9464251cdc"><code>30bf841</code></a> release(v8.1.2)</li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/44406fba5c228aba61e3ff58e9d7e41b173693d1"><code>44406fb</code></a> merge: <a href="https://github-redirect.dependabot.com/camunda/zeebe-process-test/issues/545">#545</a></li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/9f1ebf7665aacdbf43354cd9495e1c9dcfb79a4e"><code>9f1ebf7</code></a> deps(pom): bump zeebe to 8.1.2</li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/231129543b868ccbe28a8206de117ba34f47e51a"><code>2311295</code></a> release(v8.1.1): prepare for next development iteration</li> <li>See full diff in <a href="https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2">compare view</a></li> </ul> </details> <br /> Updates `zeebe-process-test-filters` from 8.1.1 to 8.1.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/camunda/zeebe-process-test/releases">zeebe-process-test-filters's releases</a>.</em></p> <blockquote> <h2>8.1.2</h2> <h2>What's Changed</h2> <ul> <li>Zeebe release 8.1.2 by <a href="https://github.com/korthout"><code>`@korthout</code></a>` in <a href="https://github-redirect.dependabot.com/camunda/zeebe-process-test/pull/545">camunda/zeebe-process-test#545</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2">https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/camunda/zeebe-process-test/commit/30bf84155a40f169f99918beb8743d9464251cdc"><code>30bf841</code></a> release(v8.1.2)</li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/44406fba5c228aba61e3ff58e9d7e41b173693d1"><code>44406fb</code></a> merge: <a href="https://github-redirect.dependabot.com/camunda/zeebe-process-test/issues/545">#545</a></li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/9f1ebf7665aacdbf43354cd9495e1c9dcfb79a4e"><code>9f1ebf7</code></a> deps(pom): bump zeebe to 8.1.2</li> <li><a href="https://github.com/camunda/zeebe-process-test/commit/231129543b868ccbe28a8206de117ba34f47e51a"><code>2311295</code></a> release(v8.1.1): prepare for next development iteration</li> <li>See full diff in <a href="https://github.com/camunda/zeebe-process-test/compare/8.1.1...8.1.2">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> 10776: Stop raft server when going inactive due to unrecoverable errors r=deepthidevaki a=deepthidevaki ## Description Previously, it was transitioning to inactive. But, in the configuration the member is still marked as active. As a result, the member transition back to active when it gets a new message from the leader. We cannot change the configuration, and mark this member as inactive because that would mean we are changing the quorum. What we really requires is that this partition is "dead" (atleast temporarily) so that it doesn't become leader again. We also don't want it to become a follower because this can also lead to partial functionality which can cause problems. For example, in follower role, raft is replicating events, but the streamprocessor or snapshotting is not working because of this error. So it is not able to compact the logs. This will eventually leads to disk space full and thus affecting other possibly healthy partitions. To fix this, in this PR we stop the raft server instead of only transitioning to inactive. The replication factor and quorum remains the same. But this node cannot become leader again until the member is restarted. ## Related issues closes #9924 10794: deps(maven): bump aws-java-sdk-core from 1.12.325 to 1.12.326 r=npepinpe a=dependabot[bot] Bumps [aws-java-sdk-core](https://github.com/aws/aws-sdk-java) from 1.12.325 to 1.12.326. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">aws-java-sdk-core's changelog</a>.</em></p> <blockquote> <h1><strong>1.12.326</strong> <strong>2022-10-21</strong></h1> <h2><strong>Amazon Cognito Identity Provider</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds a new "DeletionProtection" field to the UserPool in Cognito. Application admins can configure this value with either ACTIVE or INACTIVE value. Setting this field to ACTIVE will prevent a user pool from accidental deletion.</li> </ul> </li> </ul> <h2><strong>Amazon SageMaker Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>CreateInferenceRecommenderjob API now supports passing endpoint details directly, that will help customers to identify the max invocation and max latency they can achieve for their model and the associated endpoint along with getting recommendations on other instances.</li> </ul> </li> </ul> <h2><strong>Amazon Simple Storage Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>S3 on Outposts launches support for automatic bucket-style alias. You can use the automatic access point alias instead of an access point ARN for any object-level operation in an Outposts bucket.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java/commit/a09eee1e8f39077386c25362d4fe004294e7ca60"><code>a09eee1</code></a> AWS SDK for Java 1.12.326</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/809b864c50b846cb65f1354303a1c00f048e6dde"><code>809b864</code></a> Update GitHub version number to 1.12.326-SNAPSHOT</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-java/compare/1.12.325...1.12.326">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>
zeebe-bors-camunda bot
added a commit
that referenced
this issue
Oct 24, 2022
10776: Stop raft server when going inactive due to unrecoverable errors r=deepthidevaki a=deepthidevaki ## Description Previously, it was transitioning to inactive. But, in the configuration the member is still marked as active. As a result, the member transition back to active when it gets a new message from the leader. We cannot change the configuration, and mark this member as inactive because that would mean we are changing the quorum. What we really requires is that this partition is "dead" (atleast temporarily) so that it doesn't become leader again. We also don't want it to become a follower because this can also lead to partial functionality which can cause problems. For example, in follower role, raft is replicating events, but the streamprocessor or snapshotting is not working because of this error. So it is not able to compact the logs. This will eventually leads to disk space full and thus affecting other possibly healthy partitions. To fix this, in this PR we stop the raft server instead of only transitioning to inactive. The replication factor and quorum remains the same. But this node cannot become leader again until the member is restarted. ## Related issues closes #9924 10794: deps(maven): bump aws-java-sdk-core from 1.12.325 to 1.12.326 r=npepinpe a=dependabot[bot] Bumps [aws-java-sdk-core](https://github.com/aws/aws-sdk-java) from 1.12.325 to 1.12.326. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">aws-java-sdk-core's changelog</a>.</em></p> <blockquote> <h1><strong>1.12.326</strong> <strong>2022-10-21</strong></h1> <h2><strong>Amazon Cognito Identity Provider</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>This release adds a new "DeletionProtection" field to the UserPool in Cognito. Application admins can configure this value with either ACTIVE or INACTIVE value. Setting this field to ACTIVE will prevent a user pool from accidental deletion.</li> </ul> </li> </ul> <h2><strong>Amazon SageMaker Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>CreateInferenceRecommenderjob API now supports passing endpoint details directly, that will help customers to identify the max invocation and max latency they can achieve for their model and the associated endpoint along with getting recommendations on other instances.</li> </ul> </li> </ul> <h2><strong>Amazon Simple Storage Service</strong></h2> <ul> <li> <h3>Features</h3> <ul> <li>S3 on Outposts launches support for automatic bucket-style alias. You can use the automatic access point alias instead of an access point ARN for any object-level operation in an Outposts bucket.</li> </ul> </li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-java/commit/a09eee1e8f39077386c25362d4fe004294e7ca60"><code>a09eee1</code></a> AWS SDK for Java 1.12.326</li> <li><a href="https://github.com/aws/aws-sdk-java/commit/809b864c50b846cb65f1354303a1c00f048e6dde"><code>809b864</code></a> Update GitHub version number to 1.12.326-SNAPSHOT</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-java/compare/1.12.325...1.12.326">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - ``@dependabot` rebase` will rebase this PR - ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it - ``@dependabot` merge` will merge this PR after your CI passes on it - ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it - ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging - ``@dependabot` reopen` will reopen this PR if it is closed - ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Oct 24, 2022
Merged
korthout
added
version:8.2.0-alpha1
npepinpe
added
the
version:8.2.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When the partition is dead to #9115, the partition is marked as dead. Since it is an unrecoverable error, it is transition to inactive. The transition to inactive is somehow cause it is triggering the failure listener again and a new transition is triggered.
See the following repeated log
To Reproduce
Trigger #9115
Expected behavior
The text was updated successfully, but these errors were encountered: