daemon: reinit health monitor on live-restore #47051
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The container may have been running without health probes for an indeterminate amount of time. The container may have become unhealthy in the interim. We should probe it sooner than in steady-state, while also giving it some leeway to recover from e.g. timed-out connections. This is easy to achieve by probing the container like a freshly-started one. The original author of health-checks came to the same conclusion; the health monitor was reinitialized on live-restored containers before v17.11.0, when health monitoring of live-restored containers was accidentally broken. Revert to the original behavior. Signed-off-by: Cory Snider <csnider@mirantis.com>
corhere
added
status/1-design-review
kind/enhancement
vvoland
approved these changes
Jan 11, 2024
|
Do we have a test for this in live-restore? |
Update the TestDaemonRestartKilContainers integration test to assert that a container's healthcheck status is always reset to the Starting state after a daemon restart, even when the container is live-restored. Signed-off-by: Cory Snider <csnider@mirantis.com>
cpuguy83
approved these changes
Jan 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The container may have been running without health probes for an indeterminate amount of time. The container may have become unhealthy in the interim. We should probe it sooner than in steady-state, while also giving it some leeway to recover from e.g. timed-out connections. This is easy to achieve by probing the container like a freshly-started one. The original author of health-checks came to the same conclusion; the health monitor was reinitialized on live-restored containers before v17.11.0, when health monitoring of live-restored containers was accidentally broken. When the regression was noticed (for v23.0.0, five years after it was broken!) health monitoring of live-restored containers was resumed, not reinitialized. With the new addition of
Healthcheck.StartInterval, the original behaviour is even more compelling.- What I did
Revert to the original behavior of reinitializing the health monitor on live restore so that restored containers' health monitors are reset to the start period.
- How I did it
- How to verify it
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)