Probe components health when partition is probed #4764
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not.
korthout
force-pushed
the
4723-failed-stream-processing
branch
from
e79a87c
to
a22838e
Compare
|
I've also looked into why the ZeebePartition was unaware of the streamprocessor failing, while it is supposed to inform the ZeebePartition's CriticalComponentHealthMonitor when it fails. The reason for this is that the ZeebePartition's CriticalComponentHealthMonitor is not yet registered as the failure listener of the StreamProcessor when the failure occurs. It's visible in the following logging (which I've produced with some additional log statements): Here we see there are 2 attempts to call failureListener.onFailure(). At both moments the failureListener is null, even though the task to add the failure listener is already scheduled. That task is never executed because the actor is closed as soon as a failure occurs. If the actor would have executed the task to add the failure listener, we would have seen a log entry stating: In normal situations, where the StreamProcessor was able to fully startup successfully, it would add the failure listener and is then able to inform the monitor. In situations where it is unable to startup, it can't and we need to rely on the monitor to check-in on the streamprocessor's health using its periodic check. |
|
@deepthidevaki This change seems to make the integration test |
|
@deepthidevaki After our work this morning on the flaky test, we've combined the onFailure and handleFailure methods of StreamProcessor and moved the initial healthCheckTick in the StreamProcessor to before completing the openFuture. Injecting the failure (like in the original issue) now results in the following logging: Here we can see that the after failing, the actor is closed but because the failure listener is still null, the health monitor checking the stream processor is not informed about the stream processor becoming unhealthy. After 1 minute the CriticalComponentsHealthMonitor of the ZeebePartition probes its components, among them the StreamProcessor and discovers that the actor is closed and thus it has become unhealthy. I would like to consider whether we can improve this. Should we perhaps inform the failure listener of the current status, as soon as it registers? |
The healthcheck tick should be performed before the open future is completed, to make sure it is initialized when the health monitoring starts. This commit also changes how failures are handled. Before this commit they were handled internally in the class and rethrown. The unhandled thrown exception was then handled by the Actor which in turn calls the handleFailure method. This lead to an error message where the actor failed in phase FAILED, while it actually failed in phase STARTED and then the actor was failed afterwhich the failure was handled again.
|
@deepthidevaki I've rerun the pipeline about 10 times. There was only 1 build flaky, but this was because of another unstable test (already described here #4430). I think this integration test is no longer flaky. Please have another look at the changes :) |
deepthidevaki
approved these changes
Jun 22, 2020
|
bors r+ |
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 22, 2020
4764: Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
Build failed |
|
bors retry |
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 22, 2020
4764: Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
Build failed |
|
bors retry |
Build succeeded |
This was referenced Jun 23, 2020
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 24, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 24, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 24, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 24, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 24, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 25, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 25, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 25, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
zeebe-bors bot
added a commit
that referenced
this pull request
Jun 25, 2020
4796: [BACKPORT] Probe components health when partition is probed r=korthout a=korthout ## Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. ## Related issues closes #4723 backport of #4764 Co-authored-by: Nico korthout <nico.korthout@camunda.com>
korthout
added a commit
that referenced
this pull request
Jun 25, 2020
…backport-0.23 [BACKPORT] Probe components health when partition is probed Description When the BrokerHealthCheckService probes the ZeebePartitions for their health, the ZeebePartition only reported its own health-status. However, we cannot be sure that this status is updated when one of the components fails. To make sure that the health check is correct, it is setup to be bi-directional, so when probing, the partition should also probe its own critical components. Lastly, when this results in an UNHEALTHY state, the metrics should be updated. So we call updateHealthStatus when the partitions own state is HEALTHY, but its critical components are not. Related issues closes #4723 backport of #4764
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When the BrokerHealthCheckService probes the ZeebePartitions for their
health, the ZeebePartition only reported its own health-status. However,
we cannot be sure that this status is updated when one of the components
fails. To make sure that the health check is correct, it is setup to be
bi-directional, so when probing, the partition should also probe its own
critical components.
Lastly, when this results in an UNHEALTHY state, the metrics should be
updated. So we call updateHealthStatus when the partitions own state is
HEALTHY, but its critical components are not.
Related issues
closes #4723